# Developing-Heart-Local-Improved - Cardiomyocytes

**Pages:** 38

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/CellOracle/CellOracle_no2_vCM-Left_compact-common-TF_in-silico-perturbation.html

**Contents:**
- Import libraries¶
- Make out directory¶
- Load oracle object¶
- In silico TF perturbation analysis¶
- Calculating pseudotime gradient vectors¶
- Perturbation¶

ref: https://morris-lab.github.io/CellOracle.documentation/notebooks/04_Network_analysis/Network_analysis_with_Paul_etal_2015_data.html

**Examples:**

Example 1 (unknown):
```unknown
# Parameters
working_dir = "/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/CO_in-silico-perturbation_papermill"
path_to_oracle_object = "/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.celloracle.oracle"
path_to_oracle_links = "/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/vCM_onlyLeft_links.celloracle.links"
n_grid = 40
min_mass = 0.011
path_to_TFs = "/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/TF_selection_df_compact-common-up_2Feb2024.csv"
path_to_res_dict = "/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/vCM_onlyLeft_compact-common_silico_perturbation_2Feb2024.pkl"
path_to_plots = "/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/CO_in-silico-perturbation_papermill/figures_pm/compact-common_2Feb2024"
```

Example 2 (python):
```python
import os
import sys

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scanpy as sc
import seaborn as sns
```

Example 3 (python):
```python
import celloracle as co
co.__version__
```

Example 4 (python):
```python
/nfs/team205/kk18/miniconda3/envs/celloracle_env/lib/python3.8/site-packages/loompy/bus_file.py:68: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def twobit_to_dna(twobit: int, size: int) -> str:
/nfs/team205/kk18/miniconda3/envs/celloracle_env/lib/python3.8/site-packages/loompy/bus_file.py:85: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def dna_to_twobit(dna: str) -> int:
/nfs/team205/kk18/miniconda3/envs/celloracle_env/lib/python3.8/site-packages/loompy/bus_file.py:102: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def twobit_1hamming(twobit: int, size: int) -> List[int]:
```

Example 5 (python):
```python
WARNING:matplotlib.font_manager:Matplotlib is building the font cache; this may take a moment.
```

Example 6 (python):
```python
import session_info
session_info.show()
```

Example 7 (python):
```python
-----
celloracle          0.14.0
matplotlib          3.6.3
numpy               1.24.3
pandas              1.5.3
scanpy              1.9.5
seaborn             0.13.0
session_info        1.0.0
-----
```

Example 8 (python):
```python
Bio                         1.81
PIL                         10.0.1
anndata                     0.9.2
anyio                       NA
appdirs                     1.4.4
arrow                       1.3.0
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.13.0
backcall                    0.2.0
beta_ufunc                  NA
binom_ufunc                 NA
biothings_client            0.3.0
bottleneck                  1.3.5
brotli                      NA
certifi                     2023.07.22
cffi                        1.15.1
charset_normalizer          2.0.4
colorama                    0.4.6
comm                        0.1.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.8.0
decorator                   5.1.1
defusedxml                  0.7.1
diskcache                   5.6.3
exceptiongroup              1.1.3
executing                   2.0.0
fastjsonschema              NA
filelock                    3.12.4
fqdn                        NA
genomepy                    0.16.1
gimmemotifs                 0.17.0
goatools                    1.3.9
h5py                        3.10.0
hypergeom_ufunc             NA
idna                        3.4
igraph                      0.10.8
importlib_metadata          NA
importlib_resources         NA
invgauss_ufunc              NA
ipykernel                   6.25.2
ipywidgets                  8.1.1
isoduration                 NA
iteround                    NA
jedi                        0.19.1
jinja2                      3.1.2
joblib                      1.2.0
json5                       NA
jsonpointer                 2.4
jsonschema                  4.19.1
jsonschema_specifications   NA
jupyter_events              0.8.0
jupyter_server              2.8.0
jupyterlab_server           2.25.0
kiwisolver                  1.4.4
llvmlite                    0.41.0
logomaker                   NA
loguru                      0.7.2
loompy                      3.0.7
louvain                     0.8.0
markupsafe                  2.1.3
mkl                         2.4.0
mpl_toolkits                NA
mygene                      3.2.2
mysql                       NA
natsort                     8.4.0
nbformat                    5.9.2
nbinom_ufunc                NA
ncf_ufunc                   NA
nct_ufunc                   NA
ncx2_ufunc                  NA
networkx                    3.1
norns                       NA
numba                       0.58.0
numexpr                     2.8.4
numpy_groupies              0.9.22
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
platformdirs                3.10.0
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.6
ptyprocess                  0.7.0
pure_eval                   0.2.2
pyarrow                     13.0.0
pybedtools                  0.9.1
pycparser                   2.21
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pyfaidx                     0.7.2.2
pygments                    2.16.1
pyparsing                   3.0.9
pysam                       0.20.0
pythonjsonlogger            NA
pytz                        2023.3.post1
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
scipy                       1.10.1
send2trash                  NA
setuptools                  68.2.2
six                         1.16.0
skewnorm_ufunc              NA
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
stack_data                  0.6.3
statsmodels                 0.14.0
texttable                   1.7.0
threadpoolctl               2.2.0
tornado                     6.3.3
tqdm                        4.66.1
traitlets                   5.11.2
typing_extensions           NA
uri_template                NA
urllib3                     1.26.16
velocyto                    0.17.17
wcwidth                     0.2.8
webcolors                   1.13
websocket                   1.6.4
xdg                         NA
xgboost                     2.0.0
xxhash                      NA
yaml                        6.0.1
zipp                        NA
zmq                         25.1.1
```

Example 9 (unknown):
```unknown
-----
IPython             8.12.3
jupyter_client      8.4.0
jupyter_core        5.4.0
jupyterlab          4.0.7
notebook            7.0.6
-----
Python 3.8.18 (default, Sep 11 2023, 13:40:15) [GCC 11.2.0]
Linux-5.4.0-169-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-02-03 00:23
```

Example 10 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 11 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 12 (python):
```python
if not os.path.exists(path_to_plots):
    # Create the directory
    os.makedirs(path_to_plots)
    print("Directory created successfully")
else:
    print("Directory already exists")
```

Example 13 (unknown):
```unknown
Directory created successfully
```

Example 14 (unknown):
```unknown
# Load
oracle = co.load_hdf5(path_to_oracle_object)
```

Example 15 (python):
```python
n_cell = oracle.adata.shape[0]
print(f"cell number is :{n_cell}")

k = int(0.025*n_cell)
print(f"Auto-selected k is :{k}")
```

Example 16 (unknown):
```unknown
cell number is :16451
Auto-selected k is :411
```

Example 17 (python):
```python
%matplotlib inline
# Check clustering data
sc.pl.draw_graph(oracle.adata, color='fine_grain_time-binned')
```

Example 18 (python):
```python
# Read data from txt file
df = pd.read_csv(path_to_TFs)
list_of_TFs = list(df['TF'])

# select TFs which are in the oracle object
list_of_TFs = [x for x in list_of_TFs if x in oracle.adata.var_names]

print(len(list_of_TFs))
list_of_TFs[:5]
```

Example 19 (unknown):
```unknown
['ARID3A', 'ATF1', 'BACH2', 'BHLHE40', 'CUX1']
```

Example 20 (python):
```python
# Check gene expressions
# will be saved to a separated file
sc.settings.set_figure_params(dpi=80)
sc.pl.draw_graph(oracle.adata, color=list_of_TFs, ncols=6,
                 layer="imputed_count", use_raw=False, cmap="RdPu",
                 show=False, save='_TF_expression.png'
                )
```

Example 21 (unknown):
```unknown
WARNING: saving figure to file figures/draw_graph_fa_TF_expression.png
```

Example 22 (unknown):
```unknown
[<AxesSubplot: title={'center': 'ARID3A'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ATF1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'BACH2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'BHLHE40'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'CUX1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ESRRA'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ESRRB'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'FOXN3'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'FOXO1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'GATA4'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'GATA6'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'GTF2IRD1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'HAND2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'HEY2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'HIF1A'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'HMBOX1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'KLF12'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'KLF6'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'LCORL'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'MAX'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'MAZ'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'MBNL2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'MEF2A'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'MEF2C'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'MEF2D'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'MEIS2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'MXI1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'NCOA1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'NFATC2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'NFE2L1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'NFIA'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'NFIB'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'NFIC'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'NPAS2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'NR3C1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'NR4A1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'PBX1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'PLAG1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'PPARA'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'PRDM16'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'RORA'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'SMAD3'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'SREBF2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'TBX20'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'TFCP2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'TGIF2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'THRA'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'USF2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'YY1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ZBTB16'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ZBTB20'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ZNF148'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ZNF280D'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ZNF654'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ZSCAN18'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'AHCTF1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'CREB1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'FOXK1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'PHF21A'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'SOX5'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'UBP1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ZNF141'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ELF2'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'POU2F1'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'ZNF586'}, xlabel='FA1', ylabel='FA2'>,
 <AxesSubplot: title={'center': 'IRX3'}, xlabel='FA1', ylabel='FA2'>]
```

Example 23 (python):
```python
from celloracle.applications import Gradient_calculator
# Instantiate Gradient calculator object
gradient = Gradient_calculator(oracle_object=oracle, pseudotime_key="t")
gradient.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=200)
gradient.calculate_mass_filter(min_mass=min_mass, plot=True)

# convert the pseudotime data into grid points.
# For this calculation we can chose one of two methods.
gradient.transfer_data_into_grid(args={"method": "knn", "n_knn":50})

# Calculate graddient
gradient.calculate_gradient()
# Show results
scale_dev = 40
gradient.visualize_results(scale=scale_dev, s=5)
# Visualize results
fig, ax = plt.subplots(figsize=[6, 6])
gradient.plot_dev_flow_on_grid(scale=scale_dev, ax=ax)
```

Example 24 (python):
```python
from celloracle.applications import Oracle_development_module

# not showing plots
matplotlib.use('Agg')

# Get cell index list for the cells of interest
clusters = ['VentricularCardiomyocytesLeftCompact_t1',
 'VentricularCardiomyocytesLeftCompact_t2',
 'VentricularCardiomyocytesLeftCompact_t3']
cell_idx = np.where(oracle.adata.obs["fine_grain_time-binned"].isin(clusters))[0]

# Check index
print(cell_idx)

dev_dict = {}
for goi in list_of_TFs:
    print(goi)
    
    # Enter perturbation conditions to simulate signal propagation after the perturbation.
    oracle.simulate_shift(perturb_condition={goi: 0.0}, # this is KO simulation so expression=0
                          n_propagation=3)
    
    # Get transition probability
    oracle.estimate_transition_prob(n_neighbors=200,
                                    knn_random=True,
                                    sampled_fraction=1)

    # Calculate embedding
    oracle.calculate_embedding_shift(sigma_corr=0.05)
    
    # prepare grid
    oracle.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=200)
    oracle.calculate_mass_filter(min_mass=min_mass, plot=False)
    
    # plot vector field
    fig, ax = plt.subplots(1, 2,  figsize=[13, 6])
    scale_simulation = 0.1
    # Show quiver plot
    oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax[0])
    ax[0].set_title(f"Simulated cell identity shift vector: {goi} KO")
    # Show quiver plot that was calculated with randomized graph.
    oracle.plot_simulation_flow_random_on_grid(scale=scale_simulation, ax=ax[1])
    ax[1].set_title(f"Randomized simulation vector")
    plt.savefig(f'{path_to_plots}/{goi}_simulation_vector.pdf')
    
    # Plot vector field with cell cluster
    fig, ax = plt.subplots(figsize=[8, 8])
    oracle.plot_cluster_whole(ax=ax, s=10)
    oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax, show_background=False)
    plt.savefig(f'{path_to_plots}/{goi}_simulation_vector_with-clusters.pdf')
    
    
    dev = Oracle_development_module()
    # Load development flow
    dev.load_differentiation_reference_data(gradient_object=gradient)

    # Load simulation result
    dev.load_perturb_simulation_data(oracle_object=oracle,
                                     cell_idx_use=cell_idx, # Enter cell id list
                                     name="Lineage_Compact" # Name of this cell group. You can enter any name.
                                     )

    # Calculation
    dev.calculate_inner_product()
    dev.calculate_digitized_ip(n_bins=10)
    
    # save to the dictionary
    dev_dict[goi] = dev.copy()
    
    
    # Let's visualize the results
    vm = 0.01
    dev.visualize_development_module_layout_0(s=5,
                                              scale_for_simulation=scale_simulation,
                                              s_grid=50,
                                              scale_for_pseudotime=scale_dev,
                                              vm=vm)
    plt.savefig(f'{path_to_plots}/{goi}_results.pdf')
```

Example 25 (unknown):
```unknown
[    0     2     4 ... 16447 16448 16450]
ARID3A
```

Example 26 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 27 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 28 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 29 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 30 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 31 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 32 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 33 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 34 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 35 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 36 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 37 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 38 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 39 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 40 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 41 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 42 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 43 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 44 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 45 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 46 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 47 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 48 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 49 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 50 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 51 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 52 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 53 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 54 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 55 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 56 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 57 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 58 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 59 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 60 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 61 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 62 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 63 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 64 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 65 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 66 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 67 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 68 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 69 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 70 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 71 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 72 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 73 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 74 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 75 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 76 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 77 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 78 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 79 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 80 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 81 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 82 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 83 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 84 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 85 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 86 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 87 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 88 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 89 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 90 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 91 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 92 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 93 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 94 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 95 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 96 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 97 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 98 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 99 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 100 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 101 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 102 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 103 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 104 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 105 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 106 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 107 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 108 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 109 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 110 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 111 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 112 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 113 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 114 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 115 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 116 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 117 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 118 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 119 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 120 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 121 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 122 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 123 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 124 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 125 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 126 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 127 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 128 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 129 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 130 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 131 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 132 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 133 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 134 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 135 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 136 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 137 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 138 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 139 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 140 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 141 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 142 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 143 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 144 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 145 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 146 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 147 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 148 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 149 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 150 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 151 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 152 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 153 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 154 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 155 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 156 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 157 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 158 (unknown):
```unknown
# save
utils.save_pkl(dev_dict, path_to_res_dict)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/TransmuralAxis/tm_axis_gene.html

**Contents:**
- Import modules¶
- Functions¶
- Read in adata¶
- Add axis¶
- Differential test¶
- Slect genes to plot¶
- Pathway analysis¶
- Plot genes¶

**Examples:**

Example 1 (python):
```python
import os
import numpy as np
import scanpy as sc
import anndata
import pandas as pd
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 
import matplotlib.pyplot as plt
import seaborn as sns

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter("ignore", UserWarning)
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
anndata             0.9.2
matplotlib          3.7.2
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                         10.0.0
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.11.17
cffi                        1.15.1
charset_normalizer          3.2.0
colorama                    0.4.6
comm                        0.1.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.8
decorator                   5.1.1
defusedxml                  0.7.1
executing                   1.2.0
fastjsonschema              NA
fqdn                        NA
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_metadata          NA
importlib_resources         NA
ipykernel                   6.25.1
isoduration                 NA
jedi                        0.19.0
jinja2                      3.1.2
joblib                      1.3.2
json5                       NA
jsonpointer                 2.0
jsonschema                  4.19.0
jsonschema_specifications   NA
jupyter_events              0.7.0
jupyter_server              2.7.1
jupyterlab_server           2.24.0
kiwisolver                  1.4.5
leidenalg                   0.10.1
llvmlite                    0.40.1
markupsafe                  2.1.3
mpl_toolkits                NA
natsort                     8.4.0
nbformat                    5.9.2
numba                       0.57.1
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.16.1
pyparsing                   3.0.9
pythonjsonlogger            NA
pytz                        2023.3
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
ruamel                      NA
scipy                       1.10.1
send2trash                  NA
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
sphinxcontrib               NA
stack_data                  0.6.2
statsmodels                 0.14.0
texttable                   1.6.7
threadpoolctl               3.2.0
tornado                     6.3.3
traitlets                   5.9.0
typing_extensions           NA
uri_template                NA
urllib3                     2.0.4
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.2
yaml                        6.0.1
zipp                        NA
zmq                         25.1.1
zstandard                   0.19.0
```

Example 5 (unknown):
```unknown
-----
IPython             8.12.0
jupyter_client      8.3.0
jupyter_core        4.12.0
jupyterlab          4.0.5
notebook            7.0.2
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-5.4.0-171-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-02-22 22:02
```

Example 6 (python):
```python
import importlib.util
import sys

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/tissueaxis.py")
tissueaxis = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = tissueaxis
spec.loader.exec_module(tissueaxis)
```

Example 7 (python):
```python
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 8 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 9 (python):
```python
# This function selects the data for multiple slides from the spatial anndata object.
# Modified `select_slide` in cell2loc for multiple slides

# @param adata: anndata object with multiple spatial experiments
# @param s: list of selected slides
# @param s_col: column in adata.obs listing experiment name for each location

def select_multislide(adata, s, s_col='sample'):

    slides = adata[adata.obs[s_col].isin(s), :]
    s_keys = list(slides.uns['spatial'].keys())
    
    uns_dict = {}
    for ss in s:
        s_spatial=np.array(s_keys)[[ss in k for k in s_keys]][0]
        uns_dict[s_spatial]=slides.uns['spatial'][s_spatial]
    
    slides.uns['spatial'] = uns_dict

    return slides
```

Example 10 (python):
```python
adata_oct_healthy = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT_foetal_raw.h5ad')
print(adata_oct_healthy.shape)
adata_oct_healthy_LV = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT-LV_foetal_raw.h5ad')
print(adata_oct_healthy_LV.shape)
```

Example 11 (unknown):
```unknown
(14261, 33538)
(6767, 33538)
```

Example 12 (python):
```python
# concatenate oct adata
adata_oct = adata_oct_healthy.concatenate(
    [adata_oct_healthy_LV],
    batch_key=None,
    index_unique=None,
    join="inner"
)
```

Example 13 (python):
```python
# log-normalise
sc.pp.normalize_total(adata_oct, target_sum=1e4)
sc.pp.log1p(adata_oct)

# sc.pp.normalize_total(adata_ffpe, target_sum=1e4)
# sc.pp.log1p(adata_ffpe)

# reset var_names
adata_oct.var.set_index('SYMBOL',inplace=True)
# adata_ffpe.var.set_index('SYMBOL',inplace=True)
```

Example 14 (unknown):
```unknown
!ls /nfs/team205/heart/visium_axis/foetal/transmural_LV
```

Example 15 (unknown):
```unknown
BHF_F_Hea13295830.csv	HCAHeartST10317186.csv	HCAHeartST13162330.csv
BHF_F_Hea13295831.csv	HCAHeartST11605165.csv	HCAHeartST13162331.csv
HCAHeartST10317184.csv	HCAHeartST11605167.csv	HCAHeartST13162336.csv
HCAHeartST10317185.csv	HCAHeartST11605168.csv
```

Example 16 (python):
```python
# add
oct_samples = list(set(adata_oct.obs['sample']))

adata_oct.obs['transmural_axis'] = np.nan

axis_files = os.listdir('/nfs/team205/heart/visium_axis/foetal/transmural_LV')
for file in axis_files:
    # read in
    axis = pd.read_csv(f'/nfs/team205/heart/visium_axis/foetal/transmural_LV/{file}',index_col=0)
    # add to anndata
    if file.split('.csv')[0] in oct_samples:
        adata_oct.obs.loc[axis.index,'transmural_axis'] = axis.copy()

# select spots with axis
adata_oct = adata_oct[adata_oct.obs['transmural_axis'].isna()==False]
```

Example 17 (python):
```python
# make bin
adata_oct.obs['transmural_axis_bin'] = pd.cut(adata_oct.obs['transmural_axis'],bins=15)
```

Example 18 (python):
```python
# annotate bin, OCT
adata_oct.obs['tm_bin_annotations'] = adata_oct.obs['transmural_axis_bin'].astype('str').copy()
adata_oct.obs.replace({'tm_bin_annotations':{
    '(-1.002, -0.867]':'outer',
    '(-0.867, -0.733]':'outer',
    '(-0.733, -0.6]':'outer',
    '(-0.6, -0.467]':'outer',
    '(-0.467, -0.333]':'outer',
    '(-0.333, -0.2]':'middle',
    '(-0.2, -0.0667]':'middle',
    '(-0.0667, 0.0667]':'middle',
    '(0.0667, 0.2]':'middle',
    '(0.2, 0.333]':'middle',
    '(0.333, 0.467]':'inner',
    '(0.467, 0.6]':'inner',
    '(0.6, 0.733]':'inner',
    '(0.733, 0.867]':'inner',
    '(0.867, 1.0]':'inner',
}}, inplace=True)

set(adata_oct.obs['tm_bin_annotations'])
```

Example 19 (unknown):
```unknown
{'inner', 'middle', 'outer'}
```

Example 20 (python):
```python
# mean value per group
group_mean_val = utils.grouped_obs_mean(adata_oct, group_key='tm_bin_annotations')
# DE test
sc.tl.rank_genes_groups(adata_oct, 'tm_bin_annotations', method='t-test')
res = sc.get.rank_genes_groups_df(adata_oct, group=None, pval_cutoff=0.05)
# add mean gene expression
df = group_mean_val.reset_index().melt(id_vars=['SYMBOL'])
df.rename(columns={'SYMBOL':'names','variable':'group','value':'mean_gene_exp'},
          inplace=True)
diff_res = res.merge(df, how='left', on=['group','names'])
diff_res.head()
```

Example 21 (python):
```python
sns.scatterplot(data=diff_res, x='logfoldchanges',y='mean_gene_exp')
```

Example 22 (unknown):
```unknown
<Axes: xlabel='logfoldchanges', ylabel='mean_gene_exp'>
```

Example 23 (python):
```python
sns.scatterplot(data=diff_res, x='scores',y='mean_gene_exp')
```

Example 24 (unknown):
```unknown
<Axes: xlabel='scores', ylabel='mean_gene_exp'>
```

Example 25 (python):
```python
groups = ['outer','middle','inner']
logfc_thresh = 0.1
mean_exp_thresh = 0.05

gene_lists = {}
for grp in groups:
    gene_lists[f'{grp}_up'] = np.nan
    gene_lists[f'{grp}_down'] = np.nan
```

Example 26 (unknown):
```unknown
display(diff_res[diff_res['names']=='NRG1'])
```

Example 27 (unknown):
```unknown
['outer', 'middle', 'inner']
```

Example 28 (python):
```python
for grp in groups:
    # up genes
    genes = diff_res[(diff_res['group']==grp)&\
                (diff_res['logfoldchanges'] > logfc_thresh)&\
                (diff_res['mean_gene_exp'] > mean_exp_thresh)
               ]['names']
    gene_lists[f'{grp}_up'] = list(genes)
    print(f"{grp}_up: {len(gene_lists[f'{grp}_up'])}")

    # down genes
    genes = diff_res[(diff_res['group']==grp)&\
                (diff_res['logfoldchanges'] < -logfc_thresh)
                # (res['mean_gene_exp'] > mean_exp_thresh)
               ]['names']
    gene_lists[f'{grp}_down'] = list(genes)
    print(f"{grp}_down: {len(gene_lists[f'{grp}_down'])}")
```

Example 29 (unknown):
```unknown
outer_up: 837
outer_down: 2790
middle_up: 151
middle_down: 373
inner_up: 2447
inner_down: 1187
```

Example 30 (unknown):
```unknown
gene_lists.keys()
```

Example 31 (unknown):
```unknown
dict_keys(['outer_up', 'outer_down', 'middle_up', 'middle_down', 'inner_up', 'inner_down'])
```

Example 32 (python):
```python
import gseapy as gp
pd.set_option('display.max_rows', None)

geneset_list = ['MSigDB_Hallmark_2020','KEGG_2021_Human','GO_Biological_Process_2023']
key_to_sort = 'n_genes'
n_rank = 10

enr_res = {}
for key in gene_lists.keys():
    if ('up' in key) & (len(gene_lists[key])>0):
        print(f'##### {key} #####')

        enr_res[key] = gp.enrichr(gene_list=gene_lists[key],
                         gene_sets=geneset_list,
                         organism='human', # don't forget to set organism to the one you desired! e.g. Yeast
                         outdir=None, # don't write to disk
                        )

        for geneset in geneset_list:
            print(geneset)
            df = enr_res[key].results[enr_res[key].results['Gene_set'] == geneset]
            df = df[df['Adjusted P-value'] < 0.01]
            df['n_genes'] = [int(x.split('/')[0]) for x in df['Overlap']]
            df = df.sort_values(key_to_sort,ascending=False)
            # df = df.sort_values('Combined Score',ascending=False)
            display(df)

            # plot
            df = df[df['Adjusted P-value'] < 0.01]
            plt.rcParams['axes.grid'] = False
            plt.rcParams['figure.figsize'] = 4,3
            plt.barh(width=df[:n_rank][key_to_sort],
                     y=[x.split(' (')[0] for x in df[:n_rank]['Term']],
                     color='darkred')
            plt.gca().invert_yaxis()
            plt.xlabel(key_to_sort)
            plt.title(f'{key}, {geneset}')
            plt.margins(y=0.03)
            plt.savefig(f'{figdir}/tm-axis_pathways_{key}_{geneset}.pdf', bbox_inches="tight",dpi=300)
            plt.show()

        print('')
```

Example 33 (unknown):
```unknown
##### outer_up #####
MSigDB_Hallmark_2020
```

Example 34 (unknown):
```unknown
KEGG_2021_Human
```

Example 35 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 36 (unknown):
```unknown
##### middle_up #####
MSigDB_Hallmark_2020
```

Example 37 (unknown):
```unknown
KEGG_2021_Human
```

Example 38 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 39 (unknown):
```unknown
##### inner_up #####
MSigDB_Hallmark_2020
```

Example 40 (unknown):
```unknown
KEGG_2021_Human
```

Example 41 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 42 (python):
```python
key = 'inner_up'
hight = 10

tissueaxis.plot_axis(adata_oct,
              axis_col='transmural_axis',
              n_bin=15,
              feature_type='gene', # 'gene' or 'celltype'
              features_to_plot=gene_lists[key],
              features_to_remove=None,
              threshold_express_prop=None, # at quantile=0.9
              threshold_cell_abundance=None, # at quantile=0.9
              correlation=False,
              min_max_scale=True,
              row_cluster=True,
              figsize=(4,hight),
              cbar_pos=(-0.2, 0.5, 0.05, 0.18),
              cmap='viridis',
          xticklabels=False,
          yticklabels=False,
              title=None,
              return_feature = False,
                     save=f'{figdir}/tm-axis_OCT_genes_{key}.pdf',
    save_dpi=300,
             )
```

Example 43 (python):
```python
key = 'outer_up'
hight = 10 * len(gene_lists['outer_up'])/len(gene_lists['inner_up'])

tissueaxis.plot_axis(adata_oct,
              axis_col='transmural_axis',
              n_bin=15,
              feature_type='gene', # 'gene' or 'celltype'
              features_to_plot=gene_lists[key],
              features_to_remove=None,
              threshold_express_prop=None, # at quantile=0.9
              threshold_cell_abundance=None, # at quantile=0.9
              correlation=False,
              min_max_scale=True,
              row_cluster=True,
              figsize=(4,hight),
              cbar_pos=(-0.2, 0.5, 0.05, 0.18),
              cmap='viridis',
          xticklabels=False,
          yticklabels=False,
              title=None,
              return_feature = False,
                     save=f'{figdir}/tm-axis_OCT_genes_{key}.pdf',
    save_dpi=300,
             )
```

Example 44 (python):
```python
tissueaxis.plot_axis(adata_oct,
              axis_col='transmural_axis',
              n_bin=15,
              feature_type='gene', # 'gene' or 'celltype'
              features_to_plot=gene_lists[f'inner_up']+gene_lists[f'outer_up'],
              features_to_remove=None,
              threshold_express_prop=None, # at quantile=0.9
              threshold_cell_abundance=None, # at quantile=0.9
              correlation=False,
              min_max_scale=True,
              row_cluster=True,
              figsize=(4,10),
              cbar_pos=(-0.2, 0.5, 0.05, 0.18),
              cmap='viridis',
          xticklabels=False,
          yticklabels=False,
              title=None,
              return_feature = False,
                     save=f'{figdir}/tm-axis_OCT_genes.pdf',
    save_dpi=300,
             )
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/CellOracle/CellOracle_no1-2_in-silico-perturbatio-prep.html

**Contents:**
- Import libraries¶
- Make out directory¶
- Load oracle object¶
- In silico TF perturbation analysis¶
- Search for best min_mass.¶
- Calculating pseudotime gradient vectors¶
- Perturbation¶

ref: https://morris-lab.github.io/CellOracle.documentation/notebooks/04_Network_analysis/Network_analysis_with_Paul_etal_2015_data.html

**Examples:**

Example 1 (unknown):
```unknown
# Parameters
# working_dir = "/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/CO_in-silico-perturbation_papermill"
path_to_oracle_object = "/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM.celloracle.oracle"
path_to_oracle_links = "/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM_links.celloracle.links"

# path_to_TFs = 

clusters = ['AtrialCardiomyocytesLeft_t1',
 'AtrialCardiomyocytesLeft_t2',
 'AtrialCardiomyocytesLeft_t3']
cell_group_name = "AtrialCM-Left"

path_to_res_dict = f"/nfs/team205/heart/anndata_objects/Foetal/celloracle/{cell_group_name}_in-silico_perturbation_May2025_2.pkl"
path_to_plots = f"/nfs/team205/heart/anndata_objects/Foetal/celloracle/figures_pm/{cell_group_name}_May2025_2"
```

Example 2 (python):
```python
import os
import sys

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scanpy as sc
import seaborn as sns
```

Example 3 (python):
```python
import celloracle as co
co.__version__
```

Example 4 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 5 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 6 (python):
```python
if not os.path.exists(path_to_plots):
    # Create the directory
    os.makedirs(path_to_plots)
    print("Directory created successfully")
else:
    print("Directory already exists")
```

Example 7 (unknown):
```unknown
Directory already exists
```

Example 8 (unknown):
```unknown
# Load
oracle = co.load_hdf5(path_to_oracle_object)
```

Example 9 (python):
```python
n_cell = oracle.adata.shape[0]
print(f"cell number is :{n_cell}")

k = int(0.025*n_cell)
print(f"Auto-selected k is :{k}")
```

Example 10 (unknown):
```unknown
cell number is :5664
Auto-selected k is :141
```

Example 11 (python):
```python
%matplotlib inline
# Check clustering data
sc.pl.umap(oracle.adata, color='fine_grain_time-binned')
```

Example 12 (python):
```python
# Read in 
regulons = utils.read_pkl('/nfs/team205/heart/pyscenic/plus/baseGRN_forCO/AtrialCM_baseGRN_act.pkl')
list_of_TFs = list(regulons.keys())

# select TFs which are in the oracle object
list_of_TFs = [x for x in list_of_TFs if x in oracle.adata.var_names]

print(len(list_of_TFs))
list_of_TFs[:5]
```

Example 13 (unknown):
```unknown
['ZNF730', 'PHF21A', 'CREB1', 'CREB3L4', 'MXD1']
```

Example 14 (python):
```python
'''
# Check gene expressions
# will be saved to a separated file
sc.settings.set_figure_params(dpi=80)
sc.pl.draw_graph(oracle.adata, color=list_of_TFs, ncols=6,
                 layer="imputed_count", use_raw=False, cmap="RdPu",
                 show=False, save='_TF_expression.png'
                )
'''
```

Example 15 (python):
```python
'\n# Check gene expressions\n#\xa0will be saved to a separated file\nsc.settings.set_figure_params(dpi=80)\nsc.pl.draw_graph(oracle.adata, color=list_of_TFs, ncols=6,\n                 layer="imputed_count", use_raw=False, cmap="RdPu",\n                 show=False, save=\'_TF_expression.png\'\n                )\n'
```

Example 16 (unknown):
```unknown
n_grid = 40
n_neighbors = 100
```

Example 17 (python):
```python
# Check gene expression
goi = "PITX2"
sc.pl.umap(oracle.adata, color=[goi, oracle.cluster_column_name],
                 layer="imputed_count", use_raw=False, cmap="viridis")
```

Example 18 (unknown):
```unknown
# Enter perturbation conditions to simulate signal propagation after the perturbation.
oracle.simulate_shift(perturb_condition={goi: 0.0},
                      n_propagation=3)
```

Example 19 (unknown):
```unknown
# Get transition probability
oracle.estimate_transition_prob(n_neighbors=n_neighbors,
                                knn_random=True,
                                sampled_fraction=1)

# Calculate embedding
oracle.calculate_embedding_shift(sigma_corr=0.05)
```

Example 20 (unknown):
```unknown
oracle.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=n_neighbors)
```

Example 21 (unknown):
```unknown
oracle.suggest_mass_thresholds(n_suggestion=40)
```

Example 22 (unknown):
```unknown
min_mass = 31
scale_simulation = 5
```

Example 23 (python):
```python
%matplotlib inline
```

Example 24 (python):
```python
from celloracle.applications import Gradient_calculator
# Instantiate Gradient calculator object
gradient = Gradient_calculator(oracle_object=oracle, pseudotime_key="t")
gradient.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=n_neighbors)
gradient.calculate_mass_filter(min_mass=min_mass, plot=True)

# convert the pseudotime data into grid points.
# For this calculation we can chose one of two methods.
gradient.transfer_data_into_grid(args={"method": "knn", "n_knn":50})

# Calculate graddient
gradient.calculate_gradient()
# Show results
scale_dev = 40
gradient.visualize_results(scale=scale_dev, s=5)
# Visualize results
fig, ax = plt.subplots(figsize=[6, 6])
gradient.plot_dev_flow_on_grid(scale=scale_dev, ax=ax)
```

Example 25 (python):
```python
from tqdm import tqdm
```

Example 26 (unknown):
```unknown
list_of_TFs = ['PITX2']
```

Example 27 (python):
```python
from celloracle.applications import Oracle_development_module

# not showing plots
matplotlib.use('Agg')

# Get cell index list for the cells of interest
cell_idx = np.where(oracle.adata.obs["fine_grain_time-binned"].isin(clusters))[0]
# Check index
print(cell_idx)

dev_dict = {}
for goi in tqdm(list_of_TFs):
    # Enter perturbation conditions to simulate signal propagation after the perturbation.
    try:
        oracle.simulate_shift(perturb_condition={goi: 0.0}, # this is KO simulation so expression=0
                          n_propagation=3)
    except:
        print(f'skippint {goi} due to an error')
        continue
    
    # Get transition probability
    oracle.estimate_transition_prob(n_neighbors=n_neighbors,
                                    knn_random=True,
                                    sampled_fraction=1)

    # Calculate embedding
    oracle.calculate_embedding_shift(sigma_corr=0.05)
    
    # prepare grid
    oracle.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=n_neighbors)
    oracle.calculate_mass_filter(min_mass=min_mass, plot=False)
    
    # plot vector field
    fig, ax = plt.subplots(1, 2,  figsize=[13, 6])
    # Show quiver plot
    oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax[0])
    ax[0].set_title(f"Simulated cell identity shift vector: {goi} KO")
    # Show quiver plot that was calculated with randomized graph.
    oracle.plot_simulation_flow_random_on_grid(scale=scale_simulation, ax=ax[1])
    ax[1].set_title(f"Randomized simulation vector")
    plt.savefig(f'{path_to_plots}/{goi}_simulation_vector.pdf')
    
    # Plot vector field with cell cluster
    fig, ax = plt.subplots(figsize=[8, 8])
    oracle.plot_cluster_whole(ax=ax, s=10)
    oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax, show_background=False)
    plt.savefig(f'{path_to_plots}/{goi}_simulation_vector_with-clusters.pdf')
    
    
    dev = Oracle_development_module()
    # Load development flow
    dev.load_differentiation_reference_data(gradient_object=gradient)

    # Load simulation result
    dev.load_perturb_simulation_data(oracle_object=oracle,
                                     cell_idx_use=cell_idx, # Enter cell id list
                                     name=cell_group_name # Name of this cell group. You can enter any name.
                                     )

    # Calculation
    dev.calculate_inner_product()
    dev.calculate_digitized_ip(n_bins=10)
    
    # save to the dictionary
    dev_dict[goi] = dev.copy()
    
    # Let's visualize the results
    vm = 0.01
    dev.visualize_development_module_layout_0(s=5,
                                              scale_for_simulation=scale_simulation,
                                              s_grid=50,
                                              scale_for_pseudotime=scale_dev,
                                              vm=vm)
    plt.savefig(f'{path_to_plots}/{goi}_results.pdf')
```

Example 28 (unknown):
```unknown
[   1    2    4 ... 5659 5662 5663]
```

Example 29 (unknown):
```unknown
0%|          | 0/1 [00:00<?, ?it/s]2025-05-08 22:24:04,677 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-08 22:24:04,686 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
100%|██████████| 1/1 [05:59<00:00, 359.94s/it]
```

Example 30 (python):
```python
%matplotlib inline
```

Example 31 (unknown):
```unknown
dev.visualize_development_module_layout_0(s=5,
                                          scale_for_simulation=scale_simulation,
                                          s_grid=50,
                                          scale_for_pseudotime=40,
                                          vm=vm)
```

Example 32 (unknown):
```unknown
2025-05-08 21:38:15,504 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-08 21:38:15,513 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 33 (unknown):
```unknown
?dev.visualize_development_module_layout_0
```

Example 34 (unknown):
```unknown
Signature:
dev.visualize_development_module_layout_0(
    scale_for_pseudotime=30,
    scale_for_simulation=30,
    s=5,
    s_grid=20,
    vm=1,
    show_background=True,
    return_fig=False,
)
Docstring: <no docstring>
File:      /software/cellgen/team205/kk18/envs/celloracle_env/lib/python3.10/site-packages/celloracle/applications/development_module.py
Type:      method
```

Example 35 (unknown):
```unknown
oracle.calculate_p_mass(smooth=0.8, n_grid=40, n_neighbors=n_neighbors)
oracle.calculate_mass_filter(min_mass=min_mass, plot=True)
```

Example 36 (python):
```python
# plot vector field
fig, ax = plt.subplots(1, 2,  figsize=[13, 6])
scale_simulation = 5
# Show quiver plot
oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax[0])
ax[0].set_title(f"Simulated cell identity shift vector: {goi} KO")
# Show quiver plot that was calculated with randomized graph.
oracle.plot_simulation_flow_random_on_grid(scale=scale_simulation, ax=ax[1])
ax[1].set_title(f"Randomized simulation vector")
plt.show()
```

Example 37 (python):
```python
# Plot vector field with cell cluster
fig, ax = plt.subplots(figsize=[8, 8])

oracle.plot_cluster_whole(ax=ax, s=10)
oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax, show_background=False)
```

Example 38 (python):
```python
oracle.adata
```

Example 39 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 10870
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones', 't_binned', 'fine_grain_time-binned'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'n_cells', 'symbol', 'isin_top1000_var_mean_genes', 'isin_TFdict_targets', 'isin_TFdict_regulators', 'isin_actve_regulators'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors', 'log1p', 'fine_grain_time-binned_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
    layers: 'raw_count', 'normalized_count', 'imputed_count', 'simulation_input', 'simulated_count', 'delta_X'
    obsp: 'connectivities', 'distances'
```

Example 40 (python):
```python
sc.pl.umap(adata=oracle.adata, cmap="rainbow",
                color=["t"])
```

Example 41 (unknown):
```unknown
# save
utils.save_pkl(dev_dict, path_to_res_dict)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/LvR_aCM_biology_fitted_time_divergence_CHD_enrichment.html

**Contents:**
- Aims¶
- Read in data with all features¶
- Split into GEX and ATAC objects¶
- Make AUC objects¶
- Find features which are Left or Right specific¶
- GEX¶
  - Functional interpretation of Left and Right CM sigantures¶
- ATAC¶
- AUCgene¶
- AUCregion¶
  - Plot features which are specific to Left or Right aCMs¶
- Find features which increase over time¶
  - Functional interpretation of genes which increase over time across all aCMs¶
- Find features which are associated with time and are increasing over time¶
- Fit expression of features which change (increase or decrease) over time¶
- GEX¶
- ATAC¶
- AUCgene¶
- AUCregion¶

We want to demonstrate:

LvR: features which are different at all time (DE testing)

L: features which are associated and increase over time L: features which are associated and decrease over time

R: features which are associated and increase over time R: features which are associated and decrease over time

We start by definign a stricter set of L and R specific genes

**Examples:**

Example 1 (unknown):
```unknown
jctk_dir="/lustre/scratch126/cellgen/team205/jc48/jupyter/jctk"
```

Example 2 (python):
```python
import importlib.util
import sys
import types
import inspect

# ensure the `jctk_dir` path is already specified

def import_and_list_functions(module_name, file_path):
    spec = importlib.util.spec_from_file_location(module_name, file_path)
    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module
    spec.loader.exec_module(module)
    
    function_names = []
    for name, obj in inspect.getmembers(module):
        if isinstance(obj, types.FunctionType):
            function_names.append(name)
    
    print(f"Imported from {module_name}: {function_names}")
    
    return module  # Return the module object

# Import and list functions, then assign to a variable
plot = import_and_list_functions("plot", f"{jctk_dir}/jctk/plot.py")
utils = import_and_list_functions("utils", f"{jctk_dir}/jctk/utils.py")
api = import_and_list_functions("api", f"{jctk_dir}/jctk/api.py")
```

Example 3 (python):
```python
Imported from plot: ['GO_plot', 'highlight_all_categories', 'highlight_categories', 'normalized_stacked_bar_chart']
Imported from utils: ['check_update', 'compute_median_count', 'downsample_adata_proportionately', 'downsample_adata_randomly', 'lognorm_to_counts_with_progress', 'rank_genes_vs_nearest', 'read_gsheet', 'remove_small_batches', 'rename_adata_var_index', 'rules_based_annotation', 'shapiro', 'test_normality']
Imported from api: ['add_genomic_coordinates', 'getClinVargenes', 'getHGNCgroup', 'getHGNCgroup_dict']
```

Example 4 (python):
```python
import anndata

def split_adata_by_modality(adata, modality_column, modality_values):
    """
    Splits an AnnData object into multiple AnnData objects based on a given modality column.
    
    Parameters:
        adata (anndata.AnnData): The original AnnData object.
        modality_column (str): The name of the column in adata.var that contains the modality information.
        modality_values (list): List of unique modalities that you want to split the adata into.
        
    Returns:
        dict: A dictionary where the keys are the modality names and the values are the new AnnData objects.
    """
    # Initialize an empty dictionary to hold the new AnnData objects
    splitted_adatas = {}
    
    # Loop through each modality value to create a new AnnData object
    for modality in modality_values:
        # Get the indices of the features belonging to the current modality
        feature_idx = adata.var[modality_column] == modality
        
        # Create a new AnnData object with only the features belonging to the current modality
        new_adata = anndata.AnnData(X=adata[:, feature_idx].X,
                                    obs=adata.obs,
                                    var=adata.var.loc[feature_idx],
                                    obsm=adata.obsm,
                                    uns=adata.uns)
        
        # Add the new AnnData object to the dictionary
        splitted_adatas[modality] = new_adata
    
    return splitted_adatas
```

Example 5 (python):
```python
path_to_auc_gene="/nfs/team205/heart/pyscenic/plus/AUCscore/foetal_aCM_n570_eRegulon_AUC_gene.csv"
path_to_auc_region="/nfs/team205/heart/pyscenic/plus/AUCscore/foetal_aCM_n570_eRegulon_AUC_region.csv"

paired_adata_with_time_path = "/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA/aCMs_WholeSample_paired_with_time.h5ad"

temporal_ordering='age_days_smoothed_once'
```

Example 6 (python):
```python
import scanpy as sc
import pandas as pd
```

Example 7 (python):
```python
adata=sc.read(paired_adata_with_time_path)
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
adata.X.data[:5]
```

Example 10 (unknown):
```unknown
array([2., 1., 1., 1., 1.], dtype=float32)
```

Example 11 (python):
```python
sc.pl.umap(adata,color=["seg","milestones","t",temporal_ordering,"fine_grain"],ncols=2)
```

Example 12 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 13 (python):
```python
sc.pl.umap(adata,color=["fine_grain"],ncols=1)
sc.pl.umap(adata,color=[temporal_ordering,"t"],ncols=1,title="sample age")

sc.pl.umap(adata,color=["fine_grain","age_days","t"],ncols=1, legend_loc=None)
```

Example 14 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 15 (unknown):
```unknown
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 16 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 17 (python):
```python
adata.shape
```

Example 18 (unknown):
```unknown
(5664, 544641)
```

Example 19 (python):
```python
modality_column = "modality"
modality_values = ["Gene Expression", "Peaks"]
splitted_adatas = split_adata_by_modality(adata, modality_column, modality_values)

adata_gex = splitted_adatas["Gene Expression"]
print(adata_gex)

adata_atac = splitted_adatas["Peaks"]
print(adata_atac)
```

Example 20 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 36601
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
AnnData object with n_obs × n_vars = 5664 × 508040
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
```

Example 21 (python):
```python
adata_gex.X.data[:10]
```

Example 22 (unknown):
```unknown
array([2., 1., 1., 1., 1., 1., 8., 2., 1., 1.], dtype=float32)
```

Example 23 (python):
```python
adata_atac.X.data[:10]
```

Example 24 (unknown):
```unknown
array([2., 1., 1., 1., 1., 2., 2., 1., 1., 2.], dtype=float32)
```

Example 25 (python):
```python
adata_gex.layers["counts"]=adata_gex.X.copy()
adata_atac.layers["counts"]=adata_atac.X.copy()

sc.pp.normalize_total(adata_gex, target_sum=1e4)
sc.pp.normalize_total(adata_atac, target_sum=1e4)

sc.pp.log1p(adata_gex)
sc.pp.log1p(adata_atac)
```

Example 26 (python):
```python
WARNING: adata.X seems to be already log-transformed.
```

Example 27 (python):
```python
adata_gex.X.data[:10]
```

Example 28 (unknown):
```unknown
array([1.559231 , 1.0569503, 1.0569503, 1.0569503, 1.0569503, 1.0569503,
       2.7738788, 1.559231 , 1.0569503, 1.0569503], dtype=float32)
```

Example 29 (python):
```python
adata_atac.X.data[:10]
```

Example 30 (unknown):
```unknown
array([0.6963374 , 0.40759304, 0.40759304, 0.40759304, 0.40759304,
       0.6963374 , 0.6963374 , 0.40759304, 0.40759304, 0.6963374 ],
      dtype=float32)
```

Example 31 (python):
```python
# read in regulon scores
auc_gene = pd.read_csv(path_to_auc_gene, index_col=0)
print(auc_gene.shape)
auc_region = pd.read_csv(path_to_auc_region, index_col=0)
print(auc_region.shape)
```

Example 32 (unknown):
```unknown
(17302, 278)
(12964, 278)
```

Example 33 (python):
```python
# make regulon adata, AUC gene
# AUC Gene
# create anndata
adata_aucg = anndata.AnnData(
    X=auc_gene.loc[adata_gex.obs_names],
    obs=adata_gex.obs,
    var=pd.DataFrame(auc_gene.columns).set_index(0),
    obsm=adata_gex.obsm,
    uns= adata_gex.uns
)
# modify varnames
adata_aucg.var_names = [x.split('_(')[0] for x in adata_aucg.var_names]
print(adata_aucg.shape)

# AUC Region
# create anndata
adata_aucr = anndata.AnnData(
    X=auc_region.loc[adata_gex.obs_names],
    obs=adata_gex.obs,
    var=pd.DataFrame(auc_region.columns).set_index(0),
    obsm=adata_gex.obsm
)
# modify varnames
adata_aucr.var_names = [x.split('_(')[0] for x in adata_aucr.var_names]
print(adata_aucr.shape)
```

Example 34 (unknown):
```unknown
(5664, 278)
(5664, 278)
```

Example 35 (unknown):
```unknown
LEFT_or_RIGHT_features={}
```

Example 36 (unknown):
```unknown
log2fc_min=0.5
pval_cutoff=0.1
```

Example 37 (python):
```python
sc.tl.dendrogram(adata_gex,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_gex,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_gex,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
LEFT_or_RIGHT_features["LEFT_genes"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
LEFT_or_RIGHT_features["RIGHT_genes"]=df.names.tolist()
```

Example 38 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 39 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 40 (unknown):
```unknown
plot.GO_plot(gene_list=LEFT_or_RIGHT_features["RIGHT_genes"],display_table=True,
             gene_sets=['GO_Biological_Process_2021'],
             title_suffix=" AtrialCardiomyocytesRight",
            n_rank=10)
```

Example 41 (unknown):
```unknown
GO_Biological_Process_2021
```

Example 42 (unknown):
```unknown
plot.GO_plot(gene_list=LEFT_or_RIGHT_features["LEFT_genes"],display_table=True,gene_sets=['GO_Biological_Process_2023'],title_suffix=" AtrialCardiomyocytesLeft")
```

Example 43 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 44 (python):
```python
sc.tl.dendrogram(adata_atac,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_atac,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_atac,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_peaks"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_peaks"]=df.names.tolist()
```

Example 45 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 46 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 47 (unknown):
```unknown
adding 5316 features for LEFT...
adding 3871 features for RIGHT...
```

Example 48 (python):
```python
# Instead of doing scanpy DE testing, we use the results of ArchR for this step
df=pd.read_csv("./DAR_testing/L-vs-R_aCM.csv")
df=df[(df.Log2FC>log2fc_min)&(df.FDR<pval_cutoff)]
df["names"]=df["seqnames"].astype(str)+":"+df["start"].astype(str)+"_"+df["end"].astype(str)
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_peaks"]=df.names.tolist()

df=pd.read_csv("./DAR_testing/R-vs-L_aCM.csv")
df=df[(df.Log2FC>log2fc_min)&(df.FDR<pval_cutoff)]
df["names"]=df["seqnames"].astype(str)+":"+df["start"].astype(str)+"_"+df["end"].astype(str)
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_peaks"]=df.names.tolist()
```

Example 49 (unknown):
```unknown
adding 132 features for LEFT...
adding 57 features for RIGHT...
```

Example 50 (python):
```python
sc.tl.dendrogram(adata_aucg,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_aucg,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_aucg,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata_aucg,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_AUCgene"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata_aucg,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_AUCgene"]=df.names.tolist()
```

Example 51 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 52 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 53 (unknown):
```unknown
adding 11 features for LEFT...
adding 4 features for RIGHT...
```

Example 54 (python):
```python
sc.tl.dendrogram(adata_aucr,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_aucr,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_aucr,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata_aucr,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_AUCregion"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata_aucr,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_AUCregion"]=df.names.tolist()
```

Example 55 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 56 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 57 (unknown):
```unknown
adding 15 features for LEFT...
adding 0 features for RIGHT...
```

Example 58 (python):
```python
import scanpy as sc  # Import scanpy, assuming it's used for gene scoring and plotting

def plot_sided_features(adata, feature_type, associated_increasing_features_dict, cmap="RdPu", sides=["LEFT", "RIGHT", "COMMON","NON_SIDE_SPECIFIC"]):
    """
    Plots sided features for a given AnnData object and feature type.
    
    Args:
    adata (AnnData): An AnnData object containing gene expression data.
    feature_type (str): The type of feature to plot.
    associated_increasing_features_dict (dict): A dictionary containing associated increasing features.
    """
    # Copy the adata object to avoid modifying the original data
    plot_adata = adata.copy()

    # Initialize an empty list to store score names
    scores = []

    # Iterate over the sides and calculate scores for each
    for side in sides:
        # Retrieve the gene list from the dictionary
        gene_list = associated_increasing_features_dict[f"{feature_type}_{side}"]

        if gene_list:
            # Construct a unique name for the score
            score_name = f"{feature_type}_{side}_({len(gene_list)} features)"
            # Add the score name to the list
            scores.append(score_name)
            # Calculate the score using scanpy's score_genes function
            sc.tl.score_genes(plot_adata, gene_list=gene_list, score_name=score_name)

    # Plot the UMAP with the scores
    if len(scores)>0:
        sc.pl.umap(plot_adata, color=scores, ncols=3, wspace=0.8, cmap=cmap)
```

Example 59 (unknown):
```unknown
# New dictionary to store converted keys
LEFT_or_RIGHT_features_converted = {}

# Iterate through each item in LEFT_or_RIGHT_features
for key, value in LEFT_or_RIGHT_features.items():
    # Split the key on the underscore
    parts = key.split('_')
    
    # Rearrange the parts to match the desired format
    # New key format: 'genes_LEFT', 'genes_RIGHT', etc.
    new_key = f"{parts[1]}_{parts[0]}"

    # Add to the converted dictionary
    LEFT_or_RIGHT_features_converted[new_key] = value
LEFT_or_RIGHT_features_converted.keys()
```

Example 60 (unknown):
```unknown
dict_keys(['genes_LEFT', 'genes_RIGHT', 'peaks_LEFT', 'peaks_RIGHT', 'AUCgene_LEFT', 'AUCgene_RIGHT', 'AUCregion_LEFT', 'AUCregion_RIGHT'])
```

Example 61 (python):
```python
plot_sided_features(adata_gex,
                    sides=["LEFT","RIGHT"],
                    feature_type="genes",
                    associated_increasing_features_dict=LEFT_or_RIGHT_features_converted)

plot_sided_features(adata_atac,
                    sides=["LEFT","RIGHT"],
                    feature_type="peaks",
                    associated_increasing_features_dict=LEFT_or_RIGHT_features_converted)

plot_sided_features(adata_aucg,
                    sides=["LEFT","RIGHT"],
                    feature_type="AUCgene",
                    associated_increasing_features_dict=LEFT_or_RIGHT_features_converted)

plot_sided_features(adata_aucr,
                    sides=["LEFT","RIGHT"],
                    feature_type="AUCregion",
                    associated_increasing_features_dict=LEFT_or_RIGHT_features_converted)
```

Example 62 (python):
```python
print(LEFT_or_RIGHT_features_converted["AUCgene_LEFT"])
print(LEFT_or_RIGHT_features_converted["AUCregion_LEFT"])
print(LEFT_or_RIGHT_features_converted["AUCgene_RIGHT"])
```

Example 63 (unknown):
```unknown
['PITX2_+', 'ETS2_-', 'MAF_-', 'GATA5_+', 'SIX5_extended_+', 'ZNF337_+', 'PEG3_+', 'NFATC1_+', 'ZNF383_-', 'ARID3A_extended_-', 'MEF2A_-']
['GTF2IRD1_+', 'PITX2_+', 'IKZF2_+', 'NFE2L1_+', 'NFE2L2_+', 'TCF7L2_+', 'MGA_+', 'FOXJ2_+', 'MAF_-', 'STAT5B_+', 'MESP1_extended_+', 'ETS2_-', 'STAT4_+', 'ZNF337_+', 'ZNF470_+']
['ETS2_+', 'MAF_+', 'MYCN_+', 'ZNF526_+']
```

Example 64 (python):
```python
import pandas as pd
import numpy as np
from typing import Any

def bin_pseudotime(adata: Any, pseudotime_column: str, n_bins: int) -> Any:
    """
    Bin cells in an AnnData object into n_bins based on a specified pseudotime column.

    Parameters:
    adata (AnnData): The AnnData object containing single-cell data.
    pseudotime_column (str): The name of the column in adata.obs that contains pseudotime values.
    n_bins (int): The number of bins to divide the pseudotime values into.

    Returns:
    AnnData: The AnnData object with an additional column in adata.obs indicating the bin each cell belongs to.
    """
    # Calculate the percentiles that will be used as bin edges
    percentiles = [i*(100/n_bins) for i in range(1, n_bins)]
    bin_edges = np.percentile(adata.obs[pseudotime_column], percentiles)
    
    # Bin values into discrete intervals using pandas.cut
    adata.obs[f'{pseudotime_column}_bins'] = pd.cut(
        adata.obs[pseudotime_column],
        bins=np.insert(bin_edges, [0, len(bin_edges)], [adata.obs[pseudotime_column].min(), adata.obs[pseudotime_column].max()]),  # Include the extreme values of the pseudotime range
        include_lowest=True,
        right=True,
        labels=[str(i) for i in range(n_bins)]
    )
    
    # Return the modified AnnData object
    return adata
```

Example 65 (unknown):
```unknown
n_bins=2 # number of temporal bins
```

Example 66 (python):
```python
adata_gex = bin_pseudotime(adata_gex, 't', n_bins)
adata_atac = bin_pseudotime(adata_atac, 't', n_bins)

adata_gex = bin_pseudotime(adata_gex, temporal_ordering, n_bins)
adata_atac = bin_pseudotime(adata_atac, temporal_ordering, n_bins)

adata_aucg = bin_pseudotime(adata_aucg, 't', n_bins)
adata_aucr = bin_pseudotime(adata_aucg, 't', n_bins)

adata_aucg = bin_pseudotime(adata_aucr, temporal_ordering, n_bins)
adata_aucr = bin_pseudotime(adata_aucr, temporal_ordering, n_bins)

sc.pl.umap(adata_gex,color=["seg","milestones","t",temporal_ordering,"fine_grain","t_bins",f"{temporal_ordering}_bins"],ncols=2,wspace=0.8)
```

Example 67 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 68 (unknown):
```unknown
max_group=n_bins-1
max_group=str(max_group)
max_group
```

Example 69 (python):
```python
adata_gex.obs[f"{temporal_ordering}_bins"].value_counts()
```

Example 70 (unknown):
```unknown
0    2832
1    2832
Name: age_days_smoothed_once_bins, dtype: int64
```

Example 71 (python):
```python
increasing_features_dict={}

sc.tl.rank_genes_groups(adata_gex,groupby=f"{temporal_ordering}_bins",groups=[max_group],reference="0")
sc.pl.rank_genes_groups_dotplot(adata_gex,groupby=f"{temporal_ordering}_bins",n_genes=40,dendrogram=False)

df=sc.get.rank_genes_groups_df(adata_gex,group=max_group,log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
increasing_features_dict["INCREASING_genes"]=df.names.tolist()




sc.tl.rank_genes_groups(adata_atac,groupby=f"{temporal_ordering}_bins",groups=[max_group],reference="0")
sc.pl.rank_genes_groups_dotplot(adata_atac,groupby=f"{temporal_ordering}_bins",n_genes=40,dendrogram=False)

df=sc.get.rank_genes_groups_df(adata_atac,group=max_group,log2fc_min=log2fc_min)
#df=df[df.pvals_adj<pval_cutoff]
df=df[df.pvals<pval_cutoff] #pvals
increasing_features_dict["INCREASING_peaks"]=df.names.tolist()




sc.tl.rank_genes_groups(adata_aucg,groupby=f"{temporal_ordering}_bins",groups=[max_group],reference="0")
sc.pl.rank_genes_groups_dotplot(adata_aucg,groupby=f"{temporal_ordering}_bins",n_genes=40,dendrogram=False)

df=sc.get.rank_genes_groups_df(adata_aucg,group=max_group,log2fc_min=log2fc_min)
#df=df[df.pvals_adj<pval_cutoff]
df=df[df.pvals<pval_cutoff] #pvals
increasing_features_dict["INCREASING_AUCgene"]=df.names.tolist()




sc.tl.rank_genes_groups(adata_aucr,groupby=f"{temporal_ordering}_bins",groups=[max_group],reference="0")
sc.pl.rank_genes_groups_dotplot(adata_aucr,groupby=f"{temporal_ordering}_bins",n_genes=40,dendrogram=False)

df=sc.get.rank_genes_groups_df(adata_aucr,group=max_group,log2fc_min=log2fc_min)
#df=df[df.pvals_adj<pval_cutoff]
df=df[df.pvals<pval_cutoff] #pvals
increasing_features_dict["INCREASING_AUCregion"]=df.names.tolist()
```

Example 72 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
```

Example 73 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 74 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
```

Example 75 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 76 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
```

Example 77 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 78 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
```

Example 79 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 80 (unknown):
```unknown
plot.GO_plot(gene_list=increasing_features_dict["INCREASING_genes"],display_table=True,
             gene_sets=['MSigDB_Hallmark_2020'],
             title_suffix=" AtrialCardiomyocytesCommon increasing over time")
```

Example 81 (unknown):
```unknown
MSigDB_Hallmark_2020
```

Example 82 (unknown):
```unknown
increasing_features_dict
```

Example 83 (python):
```python
from tqdm import tqdm
import pandas as pd

sort_col = "A"
cutoff_col = 'fdr'
cutoff_value = 1e-1

associated_increasing_features_dict = {}

for feature in tqdm(["genes", "peaks", "AUCgene", "AUCregion"]):
    print("\n######### {} #########\n".format(feature))
    
    associated_increasing_NONsidespecific_features = []

    for side in ["LEFT", "RIGHT"]:
        # Load and sort the data
        df = pd.read_csv(f"./fitted_features/aCMs_WholeSample_scfatespseudotime_{feature}_{side}.csv", index_col=0)
        df = df.sort_values(sort_col, ascending=False).drop(columns="activator_repressor")
        print(f"Total number of {feature}_{side} before filtering: {len(df)}")
        
        # Apply cutoff and filter positive 'A' values
        associated_features = df[(df[cutoff_col] < cutoff_value) & (df["A"] > 0.5)].index.tolist()
        print(f"Number of {feature}_{side} after cutoff and 'A' filtering: {len(associated_features)}")
        
        # Filter for increasing features
        increasing_features = increasing_features_dict[f"INCREASING_{feature}"]
        print(f"Number of increasing {feature}: {len(increasing_features)}")
        
        # Find associated increasing features
        associated_increasing_features = [f for f in associated_features if f in increasing_features]
        print(f"Number of increasing {feature}_{side} after associating: {len(associated_increasing_features)}")
        
        # Filter for side-specific features
        side_specific_features = LEFT_or_RIGHT_features[f"{side}_{feature}"]
        print(f"Number of {side}-specific {feature}: {len(side_specific_features)}")
        
        # Find associated increasing side-specific features
        associated_increasing_sidespecific_features = [f for f in associated_increasing_features if f in side_specific_features]
        print(f"Number of associated increasing {side}-specific {feature}: {len(associated_increasing_sidespecific_features)}")
        
        # Find non-side-specific features
        non_side_specific_features = [f for f in associated_increasing_features if f not in side_specific_features]
        print(f"Number of associated increasing non-{side}-specific {feature}: {len(non_side_specific_features)}")
        
        # Combine non-side-specific features for later processing
        associated_increasing_NONsidespecific_features.extend(non_side_specific_features)
        
        # Store side-specific features
        associated_increasing_features_dict[f"{feature}_{side}"] = associated_increasing_sidespecific_features

    # Deduplicate the list of non-side-specific features
    non_side_specific_set = set(associated_increasing_NONsidespecific_features)
    print(f"Total number of associated increasing non-side-specific {feature}: {len(non_side_specific_set)}")

    # Store these NonSideSpecific features
    associated_increasing_features_dict[f"{feature}_NON_SIDE_SPECIFIC"] = list(non_side_specific_set)

    # Identify common features (present in both LEFT and RIGHT)
    common_features = non_side_specific_set.intersection(
        associated_increasing_features_dict[f"{feature}_LEFT"],
        associated_increasing_features_dict[f"{feature}_RIGHT"]
    )
    associated_increasing_features_dict[f"{feature}_COMMON"] = list(common_features)
    print(f"Total number of common non-side-specific {feature}: {len(common_features)}")

# Calculate and display counts for each category
counts_dict = {key: len(value) for key, value in associated_increasing_features_dict.items()}
print("\nCounts per category:")
for key, value in counts_dict.items():
    print(f"{key}: {value}")
```

Example 84 (unknown):
```unknown
0%|                                                                                                                                                                                                                     | 0/4 [00:00<?, ?it/s]
```

Example 85 (unknown):
```unknown
######### genes #########

Total number of genes_LEFT before filtering: 21922
```

Example 86 (unknown):
```unknown
25%|███████████████████████████████████████████████████▎                                                                                                                                                         | 1/4 [00:00<00:00,  7.55it/s]
```

Example 87 (unknown):
```unknown
Number of genes_LEFT after cutoff and 'A' filtering: 3626
Number of increasing genes: 1178
Number of increasing genes_LEFT after associating: 577
Number of LEFT-specific genes: 1440
Number of associated increasing LEFT-specific genes: 98
Number of associated increasing non-LEFT-specific genes: 479
Total number of genes_RIGHT before filtering: 21024
Number of genes_RIGHT after cutoff and 'A' filtering: 3439
Number of increasing genes: 1178
Number of increasing genes_RIGHT after associating: 485
Number of RIGHT-specific genes: 936
Number of associated increasing RIGHT-specific genes: 83
Number of associated increasing non-RIGHT-specific genes: 402
Total number of associated increasing non-side-specific genes: 569
Total number of common non-side-specific genes: 0

######### peaks #########

Total number of peaks_LEFT before filtering: 167061
Number of peaks_LEFT after cutoff and 'A' filtering: 4835
Number of increasing peaks: 5856
Number of increasing peaks_LEFT after associating: 113
Number of LEFT-specific peaks: 132
Number of associated increasing LEFT-specific peaks: 1
Number of associated increasing non-LEFT-specific peaks: 112
Total number of peaks_RIGHT before filtering: 166063
Number of peaks_RIGHT after cutoff and 'A' filtering: 7171
Number of increasing peaks: 5856
```

Example 88 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00,  2.87it/s]
```

Example 89 (unknown):
```unknown
Number of increasing peaks_RIGHT after associating: 166
Number of RIGHT-specific peaks: 57
Number of associated increasing RIGHT-specific peaks: 0
Number of associated increasing non-RIGHT-specific peaks: 166
Total number of associated increasing non-side-specific peaks: 262
Total number of common non-side-specific peaks: 0

######### AUCgene #########

Total number of AUCgene_LEFT before filtering: 278
Number of AUCgene_LEFT after cutoff and 'A' filtering: 83
Number of increasing AUCgene: 4
Number of increasing AUCgene_LEFT after associating: 4
Number of LEFT-specific AUCgene: 11
Number of associated increasing LEFT-specific AUCgene: 0
Number of associated increasing non-LEFT-specific AUCgene: 4
Total number of AUCgene_RIGHT before filtering: 278
Number of AUCgene_RIGHT after cutoff and 'A' filtering: 85
Number of increasing AUCgene: 4
Number of increasing AUCgene_RIGHT after associating: 4
Number of RIGHT-specific AUCgene: 4
Number of associated increasing RIGHT-specific AUCgene: 0
Number of associated increasing non-RIGHT-specific AUCgene: 4
Total number of associated increasing non-side-specific AUCgene: 4
Total number of common non-side-specific AUCgene: 0

######### AUCregion #########

Total number of AUCregion_LEFT before filtering: 278
Number of AUCregion_LEFT after cutoff and 'A' filtering: 75
Number of increasing AUCregion: 4
Number of increasing AUCregion_LEFT after associating: 3
Number of LEFT-specific AUCregion: 15
Number of associated increasing LEFT-specific AUCregion: 0
Number of associated increasing non-LEFT-specific AUCregion: 3
Total number of AUCregion_RIGHT before filtering: 278
Number of AUCregion_RIGHT after cutoff and 'A' filtering: 72
Number of increasing AUCregion: 4
Number of increasing AUCregion_RIGHT after associating: 3
Number of RIGHT-specific AUCregion: 0
Number of associated increasing RIGHT-specific AUCregion: 0
Number of associated increasing non-RIGHT-specific AUCregion: 3
Total number of associated increasing non-side-specific AUCregion: 4
Total number of common non-side-specific AUCregion: 0

Counts per category:
genes_LEFT: 98
genes_RIGHT: 83
genes_NON_SIDE_SPECIFIC: 569
genes_COMMON: 0
peaks_LEFT: 1
peaks_RIGHT: 0
peaks_NON_SIDE_SPECIFIC: 262
peaks_COMMON: 0
AUCgene_LEFT: 0
AUCgene_RIGHT: 0
AUCgene_NON_SIDE_SPECIFIC: 4
AUCgene_COMMON: 0
AUCregion_LEFT: 0
AUCregion_RIGHT: 0
AUCregion_NON_SIDE_SPECIFIC: 4
AUCregion_COMMON: 0
```

Example 90 (python):
```python
plot_sided_features(adata_gex,
                    feature_type="genes",
                    associated_increasing_features_dict=associated_increasing_features_dict)

plot_sided_features(adata_atac,
                    feature_type="peaks",
                    associated_increasing_features_dict=associated_increasing_features_dict)

plot_sided_features(adata_aucg,
                    feature_type="AUCgene",
                    associated_increasing_features_dict=associated_increasing_features_dict)

plot_sided_features(adata_aucr,
                    feature_type="AUCregion",
                    associated_increasing_features_dict=associated_increasing_features_dict)
```

Example 91 (unknown):
```unknown
LEFT_or_RIGHT_features={}
```

Example 92 (unknown):
```unknown
log2fc_min=2
pval_cutoff=0.05
```

Example 93 (python):
```python
sc.tl.dendrogram(adata_gex,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_gex,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_gex,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
LEFT_or_RIGHT_features["LEFT_genes"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
LEFT_or_RIGHT_features["RIGHT_genes"]=df.names.tolist()
```

Example 94 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 95 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 96 (python):
```python
sc.tl.dendrogram(adata_atac,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_atac,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_atac,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_peaks"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_peaks"]=df.names.tolist()
```

Example 97 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 98 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 99 (unknown):
```unknown
adding 1100 features for LEFT...
adding 414 features for RIGHT...
```

Example 100 (python):
```python
# Instead of doing scanpy DE testing, we use the results of ArchR for this step
df=pd.read_csv("./DAR_testing/L-vs-R_aCM.csv")
df=df[(df.Log2FC>log2fc_min)&(df.FDR<pval_cutoff)]
df["names"]=df["seqnames"].astype(str)+":"+df["start"].astype(str)+"_"+df["end"].astype(str)
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_peaks"]=df.names.tolist()

df=pd.read_csv("./DAR_testing/R-vs-L_aCM.csv")
df=df[(df.Log2FC>log2fc_min)&(df.FDR<pval_cutoff)]
df["names"]=df["seqnames"].astype(str)+":"+df["start"].astype(str)+"_"+df["end"].astype(str)
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_peaks"]=df.names.tolist()
```

Example 101 (unknown):
```unknown
adding 71 features for LEFT...
adding 19 features for RIGHT...
```

Example 102 (python):
```python
sc.tl.dendrogram(adata_aucg,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_aucg,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_aucg,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata_aucg,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_AUCgene"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata_aucg,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_AUCgene"]=df.names.tolist()
```

Example 103 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 104 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 105 (unknown):
```unknown
adding 0 features for LEFT...
adding 0 features for RIGHT...
```

Example 106 (python):
```python
sc.tl.dendrogram(adata_aucr,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_aucr,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_aucr,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata_aucr,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_AUCregion"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata_aucr,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_AUCregion"]=df.names.tolist()
```

Example 107 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 108 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 109 (unknown):
```unknown
adding 0 features for LEFT...
adding 0 features for RIGHT...
```

Example 110 (python):
```python
from tqdm import tqdm
import pandas as pd

sort_col="A"
cutff_col = 'fdr'
cutoff_value = 1e-1

associated_features_dict = {}

for feature in tqdm(["genes", "peaks", "AUCgene", "AUCregion"]):
    print("\n")
    print(f"######### {feature} #########")
    print("\n")
    
    associated_increasing_NONsidespecific_features = []

    for side in ["LEFT", "RIGHT"]:
        # Read and sort the CSV file
        df = pd.read_csv(f"./fitted_features/aCMs_WholeSample_scfatespseudotime_{feature}_{side}.csv", index_col=0).sort_values(sort_col,ascending=False)
        df = df.drop(columns="activator_repressor")
        print(f"Total number of {feature}_{side} before filtering: {len(df)}")
        
        associated_features = df[(df[cutff_col] < cutoff_value) & (df["A"] > 1)].index.tolist()
        print(f"Number of {feature}_{side} after cutoff and 'A' filtering: {len(associated_features)}")
        
        # Filter for side-specific features
        side_specific_features = LEFT_or_RIGHT_features[f"{side}_{feature}"]
        print(f"Number of {side}-specific {feature}: {len(side_specific_features)}")
        
        associated_sidespecific_features = [f for f in associated_features if f in side_specific_features]
        print(f"Number of associated {side}-specific {feature}: {len(associated_sidespecific_features)}")
        
        associated_features_dict[f"{feature}_{side}"] = associated_sidespecific_features

    # Read in COMMON features which are associated with pseudotime
    df = pd.read_csv(f"./fitted_features/aCMs_WholeSample_scfatespseudotime_{feature}_COMMON.csv", index_col=0).sort_values(sort_col,ascending=False)
    df = df.drop(columns="activator_repressor")
    print(f"Total number of {feature}_COMMON before filtering: {len(df)}")
    
    associated_features = df[(df[cutff_col] < cutoff_value) & (df["A"] > 1)].index.tolist()
    print(f"Number of {feature}_COMMON after cutoff and 'A' filtering: {len(associated_features)}")

    # Filter out any features which are known to be side-specific
    LEFT_side_specific_features = LEFT_or_RIGHT_features[f"LEFT_{feature}"]
    RIGHT_side_specific_features = LEFT_or_RIGHT_features[f"RIGHT_{feature}"]
    LEFT_or_RIGHT_side_specific_features=LEFT_side_specific_features+RIGHT_side_specific_features
    print(f"Number of (either) side-specific {feature}: {len(LEFT_or_RIGHT_side_specific_features)}")

    associated_non_sidespecific_features = [f for f in associated_features if f not in LEFT_or_RIGHT_side_specific_features]
    print(f"Number of associated non-side-specific {feature}: {len(associated_non_sidespecific_features)}")

    associated_features_dict[f"{feature}_COMMON"]=associated_non_sidespecific_features

# Calculate the counts for each category
counts_dict = {key: len(value) for key, value in associated_features_dict.items()}
print("\nCounts per category:")
for key, value in counts_dict.items():
    print(f"{key}: {value}")
```

Example 111 (unknown):
```unknown
0%|                                                                                                                                                                                                                     | 0/4 [00:00<?, ?it/s]
```

Example 112 (unknown):
```unknown
######### genes #########


Total number of genes_LEFT before filtering: 21922
Number of genes_LEFT after cutoff and 'A' filtering: 2519
Number of LEFT-specific genes: 209
Number of associated LEFT-specific genes: 28
Total number of genes_RIGHT before filtering: 21024
Number of genes_RIGHT after cutoff and 'A' filtering: 2584
Number of RIGHT-specific genes: 129
Number of associated RIGHT-specific genes: 29
```

Example 113 (unknown):
```unknown
25%|███████████████████████████████████████████████████▎                                                                                                                                                         | 1/4 [00:00<00:00,  4.54it/s]
```

Example 114 (unknown):
```unknown
Total number of genes_COMMON before filtering: 23953
Number of genes_COMMON after cutoff and 'A' filtering: 1081
Number of (either) side-specific genes: 338
Number of associated non-side-specific genes: 1048


######### peaks #########


Total number of peaks_LEFT before filtering: 167061
Number of peaks_LEFT after cutoff and 'A' filtering: 1122
Number of LEFT-specific peaks: 71
Number of associated LEFT-specific peaks: 0
Total number of peaks_RIGHT before filtering: 166063
Number of peaks_RIGHT after cutoff and 'A' filtering: 3933
Number of RIGHT-specific peaks: 19
Number of associated RIGHT-specific peaks: 0
```

Example 115 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00,  3.70it/s]
```

Example 116 (unknown):
```unknown
Total number of peaks_COMMON before filtering: 165795
Number of peaks_COMMON after cutoff and 'A' filtering: 0
Number of (either) side-specific peaks: 90
Number of associated non-side-specific peaks: 0


######### AUCgene #########


Total number of AUCgene_LEFT before filtering: 278
Number of AUCgene_LEFT after cutoff and 'A' filtering: 62
Number of LEFT-specific AUCgene: 0
Number of associated LEFT-specific AUCgene: 0
Total number of AUCgene_RIGHT before filtering: 278
Number of AUCgene_RIGHT after cutoff and 'A' filtering: 69
Number of RIGHT-specific AUCgene: 0
Number of associated RIGHT-specific AUCgene: 0
Total number of AUCgene_COMMON before filtering: 278
Number of AUCgene_COMMON after cutoff and 'A' filtering: 52
Number of (either) side-specific AUCgene: 0
Number of associated non-side-specific AUCgene: 52


######### AUCregion #########


Total number of AUCregion_LEFT before filtering: 278
Number of AUCregion_LEFT after cutoff and 'A' filtering: 50
Number of LEFT-specific AUCregion: 0
Number of associated LEFT-specific AUCregion: 0
Total number of AUCregion_RIGHT before filtering: 278
Number of AUCregion_RIGHT after cutoff and 'A' filtering: 58
Number of RIGHT-specific AUCregion: 0
Number of associated RIGHT-specific AUCregion: 0
Total number of AUCregion_COMMON before filtering: 278
Number of AUCregion_COMMON after cutoff and 'A' filtering: 56
Number of (either) side-specific AUCregion: 0
Number of associated non-side-specific AUCregion: 56

Counts per category:
genes_LEFT: 28
genes_RIGHT: 29
genes_COMMON: 1048
peaks_LEFT: 0
peaks_RIGHT: 0
peaks_COMMON: 0
AUCgene_LEFT: 0
AUCgene_RIGHT: 0
AUCgene_COMMON: 52
AUCregion_LEFT: 0
AUCregion_RIGHT: 0
AUCregion_COMMON: 56
```

Example 117 (python):
```python
# similar to before but we do not filter using the increasing features dictionary
from tqdm import tqdm
import pandas as pd

sort_col = "A"
cutoff_col = 'fdr'
cutoff_value = 1e-1

associated_increasing_features_dict = {}

for feature in tqdm(["genes", "peaks", "AUCgene", "AUCregion"]):
    print("\n######### {} #########\n".format(feature))
    
    associated_increasing_NONsidespecific_features = []

    for side in ["LEFT", "RIGHT"]:
        # Load and sort the data
        df = pd.read_csv(f"./fitted_features/aCMs_WholeSample_scfatespseudotime_{feature}_{side}.csv", index_col=0)
        df = df.sort_values(sort_col, ascending=False).drop(columns="activator_repressor")
        print(f"Total number of {feature}_{side} before filtering: {len(df)}")
        
        # Apply cutoff and filter positive 'A' values
        associated_features = df[(df[cutoff_col] < cutoff_value) & (df["A"] > 0.5)].index.tolist()
        print(f"Number of {feature}_{side} after cutoff and 'A' filtering: {len(associated_features)}")
        
        # Filter for increasing features
        #increasing_features = increasing_features_dict[f"INCREASING_{feature}"]
        #print(f"Number of increasing {feature}: {len(increasing_features)}")
        
        # Find associated increasing features
        #associated_increasing_features = [f for f in associated_features if f in increasing_features]
        #print(f"Number of increasing {feature}_{side} after associating: {len(associated_increasing_features)}")

        associated_increasing_features=associated_features
        
        # Filter for side-specific features
        side_specific_features = LEFT_or_RIGHT_features[f"{side}_{feature}"]
        print(f"Number of {side}-specific {feature}: {len(side_specific_features)}")
        
        # Find associated increasing side-specific features
        associated_increasing_sidespecific_features = [f for f in associated_increasing_features if f in side_specific_features]
        print(f"Number of associated increasing {side}-specific {feature}: {len(associated_increasing_sidespecific_features)}")
        
        # Find non-side-specific features
        non_side_specific_features = [f for f in associated_increasing_features if f not in side_specific_features]
        print(f"Number of associated increasing non-{side}-specific {feature}: {len(non_side_specific_features)}")
        
        # Combine non-side-specific features for later processing
        associated_increasing_NONsidespecific_features.extend(non_side_specific_features)
        
        # Store side-specific features
        associated_increasing_features_dict[f"{feature}_{side}"] = associated_increasing_sidespecific_features

    # Deduplicate the list of non-side-specific features
    non_side_specific_set = set(associated_increasing_NONsidespecific_features)
    print(f"Total number of associated increasing non-side-specific {feature}: {len(non_side_specific_set)}")

    # Store these NonSideSpecific features
    associated_increasing_features_dict[f"{feature}_NON_SIDE_SPECIFIC"] = list(non_side_specific_set)

    # Identify common features (present in both LEFT and RIGHT)
    common_features = non_side_specific_set.intersection(
        associated_increasing_features_dict[f"{feature}_LEFT"],
        associated_increasing_features_dict[f"{feature}_RIGHT"]
    )
    associated_increasing_features_dict[f"{feature}_COMMON"] = list(common_features)
    print(f"Total number of common non-side-specific {feature}: {len(common_features)}")

# Calculate and display counts for each category
counts_dict = {key: len(value) for key, value in associated_increasing_features_dict.items()}
print("\nCounts per category:")
for key, value in counts_dict.items():
    print(f"{key}: {value}")
```

Example 118 (unknown):
```unknown
0%|                                                                                                                                                                                                                     | 0/4 [00:00<?, ?it/s]
```

Example 119 (unknown):
```unknown
######### genes #########

Total number of genes_LEFT before filtering: 21922
Number of genes_LEFT after cutoff and 'A' filtering: 3626
Number of LEFT-specific genes: 209
Number of associated increasing LEFT-specific genes: 50
Number of associated increasing non-LEFT-specific genes: 3576
Total number of genes_RIGHT before filtering: 21024
Number of genes_RIGHT after cutoff and 'A' filtering: 3439
Number of RIGHT-specific genes: 129
Number of associated increasing RIGHT-specific genes: 42
Number of associated increasing non-RIGHT-specific genes: 3397
Total number of associated increasing non-side-specific genes: 4858
Total number of common non-side-specific genes: 0

######### peaks #########

Total number of peaks_LEFT before filtering: 167061
Number of peaks_LEFT after cutoff and 'A' filtering: 4835
Number of LEFT-specific peaks: 71
Number of associated increasing LEFT-specific peaks: 0
Number of associated increasing non-LEFT-specific peaks: 4835
```

Example 120 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 11.14it/s]
```

Example 121 (unknown):
```unknown
Total number of peaks_RIGHT before filtering: 166063
Number of peaks_RIGHT after cutoff and 'A' filtering: 7171
Number of RIGHT-specific peaks: 19
Number of associated increasing RIGHT-specific peaks: 0
Number of associated increasing non-RIGHT-specific peaks: 7171
Total number of associated increasing non-side-specific peaks: 11469
Total number of common non-side-specific peaks: 0

######### AUCgene #########

Total number of AUCgene_LEFT before filtering: 278
Number of AUCgene_LEFT after cutoff and 'A' filtering: 83
Number of LEFT-specific AUCgene: 0
Number of associated increasing LEFT-specific AUCgene: 0
Number of associated increasing non-LEFT-specific AUCgene: 83
Total number of AUCgene_RIGHT before filtering: 278
Number of AUCgene_RIGHT after cutoff and 'A' filtering: 85
Number of RIGHT-specific AUCgene: 0
Number of associated increasing RIGHT-specific AUCgene: 0
Number of associated increasing non-RIGHT-specific AUCgene: 85
Total number of associated increasing non-side-specific AUCgene: 104
Total number of common non-side-specific AUCgene: 0

######### AUCregion #########

Total number of AUCregion_LEFT before filtering: 278
Number of AUCregion_LEFT after cutoff and 'A' filtering: 75
Number of LEFT-specific AUCregion: 0
Number of associated increasing LEFT-specific AUCregion: 0
Number of associated increasing non-LEFT-specific AUCregion: 75
Total number of AUCregion_RIGHT before filtering: 278
Number of AUCregion_RIGHT after cutoff and 'A' filtering: 72
Number of RIGHT-specific AUCregion: 0
Number of associated increasing RIGHT-specific AUCregion: 0
Number of associated increasing non-RIGHT-specific AUCregion: 72
Total number of associated increasing non-side-specific AUCregion: 99
Total number of common non-side-specific AUCregion: 0

Counts per category:
genes_LEFT: 50
genes_RIGHT: 42
genes_NON_SIDE_SPECIFIC: 4858
genes_COMMON: 0
peaks_LEFT: 0
peaks_RIGHT: 0
peaks_NON_SIDE_SPECIFIC: 11469
peaks_COMMON: 0
AUCgene_LEFT: 0
AUCgene_RIGHT: 0
AUCgene_NON_SIDE_SPECIFIC: 104
AUCgene_COMMON: 0
AUCregion_LEFT: 0
AUCregion_RIGHT: 0
AUCregion_NON_SIDE_SPECIFIC: 99
AUCregion_COMMON: 0
```

Example 122 (python):
```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
from matplotlib import pyplot as plt
from matplotlib import cm
import numpy as np


def plot_fitted_expression(adata, gene_list, title, figsize, fname, show_colorbar=True):
    """
    Plots fitted gene expression over pseudotime for a list of genes.

    Parameters:
    adata (AnnData): Annotated data matrix of shape n_obs × n_vars. Rows correspond to cells and columns to genes.
    gene_list (list): List of genes to be plotted.
    title (str): Title of the plot.
    figsize (tuple): Figure size in the format (width, height).
    show_colorbar (bool): If True, display the color bar. Defaults to True.

    The function plots each gene's fitted expression as a function of pseudotime.
    The genes are ordered based on the pseudotime of their maximal expression.
    """
    pseudotime = adata.obs["t"].values  # Convert to numpy array
    pseudotime_reshaped = pseudotime.reshape(-1, 1)
    pseudotime_plot = np.linspace(min(pseudotime), max(pseudotime), 500)
    pseudotime_plot_reshaped = pseudotime_plot.reshape(-1, 1)
    gene_info = {}

    for gene in tqdm(gene_list, desc="Fitting expression..."):
        gene_expression = adata[:, gene].X.toarray()  # Adjust indexing based on your Anndata structure
        degrees = [1, 2, 3, 4, 5]
        best_score = -np.inf
        best_degree = 0

        for degree in degrees:
            model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
            score = cross_val_score(model, pseudotime_reshaped, gene_expression, cv=5).mean()
            if score > best_score:
                best_score = score
                best_degree = degree

        best_model = make_pipeline(PolynomialFeatures(best_degree), LinearRegression())
        best_model.fit(pseudotime_reshaped, gene_expression)
        fitted_expression_plot = best_model.predict(pseudotime_plot_reshaped)
        max_expression_idx = np.argmax(fitted_expression_plot)
        max_expression_time = pseudotime_plot[max_expression_idx]
        gene_info[gene] = {'model': best_model, 'max_time': max_expression_time}

    sorted_genes = sorted(gene_list, key=lambda gene: gene_info[gene]['max_time'], reverse=True)

    fig, ax = plt.subplots(figsize=figsize)
    cmap = cm.RdPu

    for i, gene in enumerate(sorted_genes):
        best_model = gene_info[gene]['model']
        fitted_expression_plot = best_model.predict(pseudotime_plot_reshaped)
        norm = plt.Normalize(fitted_expression_plot.min(), fitted_expression_plot.max())
        sc = plt.scatter(pseudotime_plot, np.ones_like(pseudotime_plot) * (i + 1), 
                         c=fitted_expression_plot, cmap=cmap, norm=norm, marker='s')

    if show_colorbar:
        plt.colorbar(sc, label='Fitted Gene Expression')

    # Remove the box around the plot
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['left'].set_visible(False)

    # Hide x-axis ticks and labels
    ax.set_xticks([])  # This removes the x-axis ticks
    ax.set_xlabel('')  # This removes the x-axis label

    # Adjust x-axis position
    ax.set_ylim(0.5, len(sorted_genes) + 0.5)

    plt.xlabel('Pseudotime')
    plt.ylabel('Gene')
    plt.yticks(np.arange(1, len(sorted_genes) + 1), sorted_genes, fontsize='xx-small', fontstyle='italic')
    plt.title(title)
    plt.savefig(fname=fname, bbox_inches='tight')
    plt.show()
```

Example 123 (python):
```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
from matplotlib import pyplot as plt
from matplotlib import cm
import numpy as np


def plot_fitted_expression(adata, gene_list, title, figsize, fname, show_colorbar=True):
    """
    Plots fitted gene expression over pseudotime for a list of genes.

    Parameters:
    adata (AnnData): Annotated data matrix of shape n_obs × n_vars. Rows correspond to cells and columns to genes.
    gene_list (list): List of genes to be plotted.
    title (str): Title of the plot.
    figsize (tuple): Figure size in the format (width, height).
    show_colorbar (bool): If True, display the color bar. Defaults to True.

    The function plots each gene's fitted expression as a function of pseudotime.
    The genes are ordered based on the pseudotime of their maximal expression.
    """
    pseudotime = adata.obs["t"].values  # Convert to numpy array

    # Normalize pseudotime to [0, 1] range
    pseudotime_min = min(pseudotime)
    pseudotime_max = max(pseudotime)
    pseudotime_normalized = (pseudotime - pseudotime_min) / (pseudotime_max - pseudotime_min)

    # Reshape for model fitting
    pseudotime_reshaped = pseudotime_normalized.reshape(-1, 1)
    
    # Create equally spaced values in [0, 1] for plotting
    pseudotime_plot = np.linspace(0, 1, 500)
    pseudotime_plot_reshaped = pseudotime_plot.reshape(-1, 1)
    
    gene_info = {}

    for gene in tqdm(gene_list, desc="Fitting expression..."):
        gene_expression = adata[:, gene].X.toarray()  # Adjust indexing based on your Anndata structure
        degrees = [1, 2, 3, 4, 5]
        best_score = -np.inf
        best_degree = 0

        for degree in degrees:
            model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
            score = cross_val_score(model, pseudotime_reshaped, gene_expression, cv=5).mean()
            if score > best_score:
                best_score = score
                best_degree = degree

        best_model = make_pipeline(PolynomialFeatures(best_degree), LinearRegression())
        best_model.fit(pseudotime_reshaped, gene_expression)
        fitted_expression_plot = best_model.predict(pseudotime_plot_reshaped)
        max_expression_idx = np.argmax(fitted_expression_plot)
        max_expression_time = pseudotime_plot[max_expression_idx]
        gene_info[gene] = {'model': best_model, 'max_time': max_expression_time}

    sorted_genes = sorted(gene_list, key=lambda gene: gene_info[gene]['max_time'], reverse=True)

    fig, ax = plt.subplots(figsize=figsize)
    cmap = cm.RdPu

    for i, gene in enumerate(sorted_genes):
        best_model = gene_info[gene]['model']
        fitted_expression_plot = best_model.predict(pseudotime_plot_reshaped)
        norm = plt.Normalize(fitted_expression_plot.min(), fitted_expression_plot.max())
        sc = plt.scatter(pseudotime_plot, np.ones_like(pseudotime_plot) * (i + 1), 
                         c=fitted_expression_plot, cmap=cmap, norm=norm, marker='s')

    if show_colorbar:
        plt.colorbar(sc, label='Fitted Gene Expression')

    # Remove the box around the plot
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['left'].set_visible(False)

    # If you wish, hide x-axis ticks and labels by uncommenting
    #ax.set_xticks([])  # This removes the x-axis ticks
    #ax.set_xlabel('')  # This removes the x-axis label
    
    ax.set_xlim(0, 1)
    ax.set_ylim(0.5, len(sorted_genes) + 0.5)

    plt.xlabel('Pseudotime')
    plt.ylabel('Gene')
    plt.yticks(np.arange(1, len(sorted_genes) + 1), sorted_genes, fontsize='xx-small', fontstyle='italic')
    plt.title(title)
    plt.savefig(fname=fname, bbox_inches='tight')
    plt.show()
```

Example 124 (unknown):
```unknown
associated_increasing_features_dict.keys()
```

Example 125 (unknown):
```unknown
dict_keys(['genes_LEFT', 'genes_RIGHT', 'genes_NON_SIDE_SPECIFIC', 'genes_COMMON', 'peaks_LEFT', 'peaks_RIGHT', 'peaks_NON_SIDE_SPECIFIC', 'peaks_COMMON', 'AUCgene_LEFT', 'AUCgene_RIGHT', 'AUCgene_NON_SIDE_SPECIFIC', 'AUCgene_COMMON', 'AUCregion_LEFT', 'AUCregion_RIGHT', 'AUCregion_NON_SIDE_SPECIFIC', 'AUCregion_COMMON'])
```

Example 126 (python):
```python
figsize=(2,4.5)
top_n_features=28
file_extension=".png"
file_extension=".svg"

plot_fitted_expression(adata_gex[adata_gex.obs.fine_grain=="AtrialCardiomyocytesLeft"], gene_list=associated_features_dict["genes_LEFT"][:top_n_features], title="LEFT atrial cardiomyocyte maturation", figsize=figsize, show_colorbar=False,fname=f"genes_LEFT{file_extension}")
plot_fitted_expression(adata_gex[adata_gex.obs.fine_grain=="AtrialCardiomyocytesRight"], gene_list=associated_features_dict["genes_RIGHT"][:top_n_features], title="RIGHT atrial cardiomyocyte maturation", figsize=figsize, show_colorbar=False,fname=f"genes_RIGHT{file_extension}")

#top_n_features=80
#figsize=(2,10)

plot_fitted_expression(adata_gex, gene_list=associated_features_dict["genes_COMMON"][:top_n_features], title="COMMON atrial cardiomyocyte maturation", figsize=figsize, show_colorbar=False,fname=f"genes_COMMON{file_extension}")
```

Example 127 (unknown):
```unknown
Fitting expression...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:02<00:00, 11.82it/s]
```

Example 128 (unknown):
```unknown
Fitting expression...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:01<00:00, 17.55it/s]
```

Example 129 (unknown):
```unknown
Fitting expression...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:01<00:00, 15.48it/s]
```

Example 130 (python):
```python
gene_list=associated_features_dict["genes_LEFT"]
adata_subset=adata_gex[adata_gex.obs.fine_grain=="AtrialCardiomyocytesLeft"].copy()
distance_threshold=4.5
```

Example 131 (python):
```python
import seaborn as sns
import pandas as pd
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, MinMaxScaler
from sklearn.linear_model import LinearRegression
from scipy.spatial.distance import pdist
from scipy.cluster.hierarchy import linkage, fcluster
from tqdm import tqdm
import matplotlib.pyplot as plt

def create_clustermap(adata, gene_list, n_bins, figsize, desired_n_clusters):
    """
    Creates a clustermap of fitted gene expression across pseudotime bins for a list of genes.

    Parameters:
    adata (AnnData): Annotated data matrix.
    gene_list (list): List of genes to be included.
    n_bins (int): Number of bins to divide pseudotime into.
    figsize (tuple): Figure size in the format (width, height).
    """
    # Normalize and reshape pseudotime
    pseudotime = adata.obs["t"].values
    pseudotime_min = min(pseudotime)
    pseudotime_max = max(pseudotime)
    pseudotime_normalized = (pseudotime - pseudotime_min) / (pseudotime_max - pseudotime_min)
    pseudotime_reshaped = pseudotime_normalized.reshape(-1, 1)

    # Create pseudotime bins
    bins = np.linspace(0, 1, n_bins + 1)
    pseudotime_binned = np.digitize(pseudotime_normalized, bins) - 1

    # DataFrame to store fitted expressions
    expression_data = pd.DataFrame(index=gene_list, columns=range(n_bins))

    print(f"Creating clustermap for {len(gene_list)} features across {n_bins} bins")

    for gene in tqdm(gene_list, desc="Fitting expression..."):
        gene_expression = adata[:, gene].X.toarray()

        # Check if gene_expression is empty or almost empty
        if gene_expression.size == 0 or np.count_nonzero(gene_expression) < 5:
            print(f"Insufficient data for gene: {gene}")
            continue

        # Fit the model
        model = make_pipeline(PolynomialFeatures(2), LinearRegression())
        model.fit(pseudotime_reshaped, gene_expression)

        # Predict expression for each bin
        for i in range(n_bins):
            bin_range = np.linspace(bins[i], bins[i + 1], 10).reshape(-1, 1)
            predicted_expression = model.predict(bin_range).mean()
            expression_data.loc[gene, i] = predicted_expression  # Direct assignment
    
    expression_data = expression_data.astype(float).fillna(0)

    # Perform min-max normalization
    scaler = MinMaxScaler()
    expression_data_normalized = pd.DataFrame(scaler.fit_transform(expression_data.T).T, index=gene_list, columns=expression_data.columns) # row-wise (per gene) min-max normalisation


    # Compute the distance matrix for the rows (genes) and then the linkage matrix
    distance_matrix = pdist(expression_data_normalized)
    row_linkage = linkage(distance_matrix, method='average')

    # Define clusters using the row linkage
    clusters = fcluster(row_linkage, desired_n_clusters, criterion='maxclust')

    # Define clusters using a distance threshold
    clusters = fcluster(row_linkage, distance_threshold, criterion='distance')

    # Generate a color palette with seaborn and map cluster labels to colors
    cluster_colors = sns.color_palette("tab20", n_colors=len(np.unique(clusters)))
    row_colors = pd.Series(clusters, index=gene_list).map(dict(zip(np.unique(clusters), cluster_colors)))

    # Create clustermap with row colors
    g = sns.clustermap(expression_data_normalized, row_linkage=row_linkage, figsize=figsize, cmap='RdPu', yticklabels=True, xticklabels=True, col_cluster=False, row_colors=row_colors)
    g.cax.set_visible(False)
    
    plt.setp(g.ax_heatmap.get_yticklabels(), fontsize='xx-small', fontstyle='italic')
    plt.show()

    return row_linkage, clusters

row_linkage, clusters = create_clustermap(adata=adata_subset, gene_list=gene_list, n_bins=500, figsize=(4,30), desired_n_clusters=3)




from collections import defaultdict

def create_cluster_dict(clusters, gene_list):
    """
    Creates a dictionary mapping cluster names to lists of genes, adjusting for zero-based indexing.

    Parameters:
    clusters (array): Array of cluster assignments for each gene.
    gene_list (list): List of genes corresponding to the clusters.

    Returns:
    dict: Dictionary with cluster names as keys and lists of genes as values.
    """
    cluster_dict = defaultdict(list)
    for gene, cluster in zip(gene_list, clusters):
        #print(f"Gene: {gene}, Cluster: {cluster}")  # Diagnostic print
        cluster_name=f"cluster_{cluster - 1}"
        cluster_dict[cluster_name].append(gene)

    return dict(cluster_dict)

# Example usage
cluster_dict = create_cluster_dict(clusters, gene_list=gene_list)

# Check the clusters and the number of genes in each
for cluster, genes in cluster_dict.items():
    print(f"{cluster}: {len(genes)} genes")

cluster_dict.keys()
scores=[]
for cluster_name,cluster_genes in cluster_dict.items():
    score_name=f"{cluster_name}_({len(cluster_genes)} genes)"
    scores.append(score_name)
    sc.tl.score_genes(adata_gex,gene_list=cluster_genes,score_name=f"{score_name}")

sc.pl.umap(adata_gex, color=scores, ncols=4, wspace=0.8, cmap="RdPu")








import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

optimal_orders = {}

for score in scores:
    # Prepare the data
    tmp_df = adata_gex.obs[["t", score]]
    X = tmp_df["t"].values.reshape(-1, 1)
    y = tmp_df[score].values

    # Split the data into training and testing sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    best_order = 1
    best_mse = float('inf')

    # Test different polynomial orders
    for order in range(1, 6):
        model = make_pipeline(PolynomialFeatures(order), LinearRegression())
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        mse = mean_squared_error(y_test, predictions)

        if mse < best_mse:
            best_mse = mse
            best_order = order

    optimal_orders[score] = best_order






import matplotlib.pyplot as plt
import seaborn as sns

# Number of scores (subplots)
num_scores = len(scores)

# Create a figure with a row of subplots
fig, axes = plt.subplots(1, num_scores, figsize=(5 * num_scores, 5), sharey=True)

# Track the global min and max for the y-axis
global_min, global_max = float('inf'), float('-inf')

# Plot each score
for i, score in enumerate(scores):
    # Prepare the data
    tmp_df = adata_gex.obs[["t", score, "fine_grain"]]
    
    # Plot for each fine_grain category
    fine_grain_categories = tmp_df["fine_grain"].unique()
    for category in fine_grain_categories:
        category_data = tmp_df[tmp_df["fine_grain"] == category]
        sns.regplot(ax=axes[i], data=category_data, x="t", y=score, order=optimal_orders[score], scatter=False, label=category)
    
    # Set the title and add legend
    axes[i].set_title(score)
    axes[i].legend()

    # Update global min and max
    current_min, current_max = tmp_df[score].min(), tmp_df[score].max()
    global_min, global_max = min(global_min, current_min), max(global_max, current_max)

# Set the same y-axis limits for all subplots
for ax in axes:
    ax.set_ylim(global_min, global_max)

# Adjust layout
plt.tight_layout()
plt.show()
```

Example 132 (unknown):
```unknown
Creating clustermap for 28 features across 500 bins
```

Example 133 (unknown):
```unknown
Fitting expression...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:02<00:00, 10.55it/s]
```

Example 134 (unknown):
```unknown
cluster_3: 7 genes
cluster_2: 6 genes
cluster_4: 2 genes
cluster_6: 1 genes
cluster_0: 8 genes
cluster_5: 3 genes
cluster_1: 1 genes
```

Example 135 (unknown):
```unknown
cluster_dict["cluster_3"]
```

Example 136 (unknown):
```unknown
['MBP', 'KCNMB2', 'GAN', 'SLC8A3', 'A2M', 'KCNMB2-AS1', 'KSR1']
```

Example 137 (unknown):
```unknown
gene_set_A=cluster_dict["cluster_3"]+cluster_dict["cluster_4"]+cluster_dict["cluster_6"]
gene_set_A
```

Example 138 (unknown):
```unknown
['MBP',
 'KCNMB2',
 'GAN',
 'SLC8A3',
 'A2M',
 'KCNMB2-AS1',
 'KSR1',
 'PANCR',
 'PITX2',
 'DCBLD2']
```

Example 139 (python):
```python
gene_list=associated_features_dict["genes_RIGHT"]
adata_subset=adata_gex[adata_gex.obs.fine_grain=="AtrialCardiomyocytesRight"].copy()
distance_threshold=4.5
```

Example 140 (python):
```python
import seaborn as sns
import pandas as pd
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, MinMaxScaler
from sklearn.linear_model import LinearRegression
from scipy.spatial.distance import pdist
from scipy.cluster.hierarchy import linkage, fcluster
from tqdm import tqdm
import matplotlib.pyplot as plt

def create_clustermap(adata, gene_list, n_bins, figsize, desired_n_clusters):
    """
    Creates a clustermap of fitted gene expression across pseudotime bins for a list of genes.

    Parameters:
    adata (AnnData): Annotated data matrix.
    gene_list (list): List of genes to be included.
    n_bins (int): Number of bins to divide pseudotime into.
    figsize (tuple): Figure size in the format (width, height).
    """
    # Normalize and reshape pseudotime
    pseudotime = adata.obs["t"].values
    pseudotime_min = min(pseudotime)
    pseudotime_max = max(pseudotime)
    pseudotime_normalized = (pseudotime - pseudotime_min) / (pseudotime_max - pseudotime_min)
    pseudotime_reshaped = pseudotime_normalized.reshape(-1, 1)

    # Create pseudotime bins
    bins = np.linspace(0, 1, n_bins + 1)
    pseudotime_binned = np.digitize(pseudotime_normalized, bins) - 1

    # DataFrame to store fitted expressions
    expression_data = pd.DataFrame(index=gene_list, columns=range(n_bins))

    print(f"Creating clustermap for {len(gene_list)} features across {n_bins} bins")

    for gene in tqdm(gene_list, desc="Fitting expression..."):
        gene_expression = adata[:, gene].X.toarray()

        # Check if gene_expression is empty or almost empty
        if gene_expression.size == 0 or np.count_nonzero(gene_expression) < 5:
            print(f"Insufficient data for gene: {gene}")
            continue

        # Fit the model
        model = make_pipeline(PolynomialFeatures(2), LinearRegression())
        model.fit(pseudotime_reshaped, gene_expression)

        # Predict expression for each bin
        for i in range(n_bins):
            bin_range = np.linspace(bins[i], bins[i + 1], 10).reshape(-1, 1)
            predicted_expression = model.predict(bin_range).mean()
            expression_data.loc[gene, i] = predicted_expression  # Direct assignment
    
    expression_data = expression_data.astype(float).fillna(0)

    # Perform min-max normalization
    scaler = MinMaxScaler()
    expression_data_normalized = pd.DataFrame(scaler.fit_transform(expression_data.T).T, index=gene_list, columns=expression_data.columns) # row-wise (per gene) min-max normalisation


    # Compute the distance matrix for the rows (genes) and then the linkage matrix
    distance_matrix = pdist(expression_data_normalized)
    row_linkage = linkage(distance_matrix, method='average')

    # Define clusters using the row linkage
    clusters = fcluster(row_linkage, desired_n_clusters, criterion='maxclust')

    # Define clusters using a distance threshold
    clusters = fcluster(row_linkage, distance_threshold, criterion='distance')

    # Generate a color palette with seaborn and map cluster labels to colors
    cluster_colors = sns.color_palette("tab20", n_colors=len(np.unique(clusters)))
    row_colors = pd.Series(clusters, index=gene_list).map(dict(zip(np.unique(clusters), cluster_colors)))

    # Create clustermap with row colors
    g = sns.clustermap(expression_data_normalized, row_linkage=row_linkage, figsize=figsize, cmap='RdPu', yticklabels=True, xticklabels=True, col_cluster=False, row_colors=row_colors)
    g.cax.set_visible(False)
    
    plt.setp(g.ax_heatmap.get_yticklabels(), fontsize='xx-small', fontstyle='italic')
    plt.show()

    return row_linkage, clusters

row_linkage, clusters = create_clustermap(adata=adata_subset, gene_list=gene_list, n_bins=500, figsize=(4,30), desired_n_clusters=3)




from collections import defaultdict

def create_cluster_dict(clusters, gene_list):
    """
    Creates a dictionary mapping cluster names to lists of genes, adjusting for zero-based indexing.

    Parameters:
    clusters (array): Array of cluster assignments for each gene.
    gene_list (list): List of genes corresponding to the clusters.

    Returns:
    dict: Dictionary with cluster names as keys and lists of genes as values.
    """
    cluster_dict = defaultdict(list)
    for gene, cluster in zip(gene_list, clusters):
        #print(f"Gene: {gene}, Cluster: {cluster}")  # Diagnostic print
        cluster_name=f"cluster_{cluster - 1}"
        cluster_dict[cluster_name].append(gene)

    return dict(cluster_dict)

# Example usage
cluster_dict = create_cluster_dict(clusters, gene_list=gene_list)

# Check the clusters and the number of genes in each
for cluster, genes in cluster_dict.items():
    print(f"{cluster}: {len(genes)} genes")

cluster_dict.keys()
scores=[]
for cluster_name,cluster_genes in cluster_dict.items():
    score_name=f"{cluster_name}_({len(cluster_genes)} genes)"
    scores.append(score_name)
    sc.tl.score_genes(adata_gex,gene_list=cluster_genes,score_name=f"{score_name}")

sc.pl.umap(adata_gex, color=scores, ncols=4, wspace=0.8, cmap="RdPu")








import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

optimal_orders = {}

for score in scores:
    # Prepare the data
    tmp_df = adata_gex.obs[["t", score]]
    X = tmp_df["t"].values.reshape(-1, 1)
    y = tmp_df[score].values

    # Split the data into training and testing sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    best_order = 1
    best_mse = float('inf')

    # Test different polynomial orders
    for order in range(1, 6):
        model = make_pipeline(PolynomialFeatures(order), LinearRegression())
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        mse = mean_squared_error(y_test, predictions)

        if mse < best_mse:
            best_mse = mse
            best_order = order

    optimal_orders[score] = best_order






import matplotlib.pyplot as plt
import seaborn as sns

# Number of scores (subplots)
num_scores = len(scores)

# Create a figure with a row of subplots
fig, axes = plt.subplots(1, num_scores, figsize=(5 * num_scores, 5), sharey=True)

# Track the global min and max for the y-axis
global_min, global_max = float('inf'), float('-inf')

# Plot each score
for i, score in enumerate(scores):
    # Prepare the data
    tmp_df = adata_gex.obs[["t", score, "fine_grain"]]
    
    # Plot for each fine_grain category
    fine_grain_categories = tmp_df["fine_grain"].unique()
    for category in fine_grain_categories:
        category_data = tmp_df[tmp_df["fine_grain"] == category]
        sns.regplot(ax=axes[i], data=category_data, x="t", y=score, order=optimal_orders[score], scatter=False, label=category)
    
    # Set the title and add legend
    axes[i].set_title(score)
    axes[i].legend()

    # Update global min and max
    current_min, current_max = tmp_df[score].min(), tmp_df[score].max()
    global_min, global_max = min(global_min, current_min), max(global_max, current_max)

# Set the same y-axis limits for all subplots
for ax in axes:
    ax.set_ylim(global_min, global_max)

# Adjust layout
plt.tight_layout()
plt.show()
```

Example 141 (unknown):
```unknown
Creating clustermap for 29 features across 500 bins
```

Example 142 (unknown):
```unknown
Fitting expression...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 29/29 [00:02<00:00, 10.81it/s]
```

Example 143 (unknown):
```unknown
cluster_1: 8 genes
cluster_4: 5 genes
cluster_2: 4 genes
cluster_3: 10 genes
cluster_0: 2 genes
```

Example 144 (unknown):
```unknown
gene_set_B=cluster_dict["cluster_2"]+cluster_dict["cluster_1"]
gene_set_B
```

Example 145 (unknown):
```unknown
['DAB1',
 'UST',
 'ANGPT1',
 'GNAL',
 'KIF26B',
 'KCNQ5',
 'HEY1',
 'HSPA6',
 'RFTN1',
 'MAF',
 'OTOGL',
 'PDE1A']
```

Example 146 (python):
```python
sc.pl.umap(adata_gex, color=["milestones","seg"], ncols=4, wspace=0.8)
```

Example 147 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 148 (python):
```python
import pandas as pd

# Assuming adata_gex is your AnnData object containing scRNAseq data
# and it has 't' and 'milestones' in adata_gex.obs

# Filter the rows where 'milestones' is '60'
filtered_data = adata_gex.obs[adata_gex.obs['milestones'] == '60']

# Find the maximum 't' value in this filtered group
bifurcation_point = filtered_data['t'].median()

# Output the bifurcation point
print("Bifurcation Point:", bifurcation_point)
```

Example 149 (unknown):
```unknown
Bifurcation Point: 1.9802904538320618
```

Example 150 (python):
```python
import numpy as np
import pandas as pd

# Assuming adata_gex is your AnnData object containing scRNAseq data
# and bifurcation_point is already defined as shown previously

# Define the number of bins you want
X = 4  # Adjust this value as needed

# Get the minimum 't' value from adata_gex
min_t_value = adata_gex.obs['t'].min()

# Create bins between the minimum 't' value and the bifurcation point
bins = np.linspace(min_t_value, bifurcation_point, X + 1)  # X+1 because we need X bins

# Assign each cell to a pseudotime bin or "post_bifurcation"
adata_gex.obs['pseudotime_bin'] = pd.cut(adata_gex.obs['t'], bins=bins, include_lowest=True, 
                                         labels=[f'pseudotime_bin_{i}' for i in range(1, X + 1)])
adata_gex.obs['pseudotime_bin'] = adata_gex.obs['pseudotime_bin'].cat.add_categories('post_bifurcation')
adata_gex.obs.loc[adata_gex.obs['t'] > bifurcation_point, 'pseudotime_bin'] = 'post_bifurcation'

# You can now work with the 'pseudotime_bin' column in adata_gex.obs for further analysis
sc.pl.umap(adata_gex, color=["pseudotime_bin"], ncols=1, wspace=0.8, size=50)
```

Example 151 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 152 (python):
```python
adata_gex.obs.fine_grain.value_counts()
```

Example 153 (unknown):
```unknown
AtrialCardiomyocytesLeft     3041
AtrialCardiomyocytesRight    2623
Name: fine_grain, dtype: int64
```

Example 154 (python):
```python
# Step 1: Copy values from 'fine_grain' to the new column
adata_gex.obs['AtrialCardiomyocytesImmature_annotation'] = adata_gex.obs['fine_grain']

# Add new category to the column
new_category = "AtrialCardiomyocytesImmature"
if new_category not in adata_gex.obs['AtrialCardiomyocytesImmature_annotation'].cat.categories:
    adata_gex.obs['AtrialCardiomyocytesImmature_annotation'] = adata_gex.obs['AtrialCardiomyocytesImmature_annotation'].cat.add_categories(new_category)

# Step 2: Modify values for cells not equal to "post_bifurcation" in 'pseudotime_bin'
adata_gex.obs.loc[adata_gex.obs['pseudotime_bin'] != 'post_bifurcation', 'AtrialCardiomyocytesImmature_annotation'] = new_category

# Verify the value counts
print(adata_gex.obs['AtrialCardiomyocytesImmature_annotation'].value_counts())
print("saving annotation for use with c2l...")
adata_gex.obs['AtrialCardiomyocytesImmature_annotation'].to_csv("AtrialCardiomyocytesImmature_annotation.csv")
sc.pl.umap(adata_gex,color="AtrialCardiomyocytesImmature_annotation")
```

Example 155 (unknown):
```unknown
AtrialCardiomyocytesLeft        2828
AtrialCardiomyocytesRight       2509
AtrialCardiomyocytesImmature     327
Name: AtrialCardiomyocytesImmature_annotation, dtype: int64
saving annotation for use with c2l...
```

Example 156 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 157 (python):
```python
adata_gex.X=adata_gex.layers["counts"].copy()
sc.pp.normalize_total(adata_gex, target_sum=1e4)
sc.pp.log1p(adata_gex)
```

Example 158 (python):
```python
WARNING: adata.X seems to be already log-transformed.
```

Example 159 (python):
```python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import scipy.stats
from scipy.sparse import issparse
from tqdm import tqdm
import scanpy as sc
import numpy as np

# Function to calculate the correlation matrix and scatter plot data
def calculate_correlation(bin_cells, gene_set_A, gene_set_B, gene_indices):
    combined_gene_set = gene_set_A + gene_set_B
    # Initialize an empty DataFrame for the correlation matrix
    correlation_matrix = pd.DataFrame(index=combined_gene_set, columns=combined_gene_set)
    scatter_data = []

    # Calculate correlations for each pair of genes
    for gene1 in tqdm(combined_gene_set):
        for gene2 in combined_gene_set:
            # Extract expression values for gene1 and gene2
            x_values = bin_cells.X[:, gene_indices[gene1]].toarray().flatten() if issparse(bin_cells.X) else bin_cells.X[:, gene_indices[gene1]].flatten()
            y_values = bin_cells.X[:, gene_indices[gene2]].toarray().flatten() if issparse(bin_cells.X) else bin_cells.X[:, gene_indices[gene2]].flatten()

            # Calculate Pearson correlation
            correlation, _ = scipy.stats.pearsonr(x_values, y_values)

            # Store the correlation in the matrix
            correlation_matrix.at[gene1, gene2] = correlation

    # Calculate mean correlation for each gene with gene_set_A and gene_set_B
    for gene in combined_gene_set:
        mean_corr_with_A = correlation_matrix.loc[gene, gene_set_A].mean()
        mean_corr_with_B = correlation_matrix.loc[gene, gene_set_B].mean()
        gene_set_label = 'Left aCM early module' if gene in gene_set_A else 'Right aCM early module'

        scatter_data.append({'gene': gene, 'mean_corr_with_A': mean_corr_with_A, 'mean_corr_with_B': mean_corr_with_B, 'gene_set': gene_set_label})

    scatter_df = pd.DataFrame(scatter_data)

    return scatter_df

# Function to add labels to the genes closest to the bottom right and top left of each scatter plot
def label_extreme_genes(scatter_df, ax):
    # Define bottom right and top left points
    bottom_right_point = (1, -1)
    top_left_point = (-1, 1)

    # Function to calculate Euclidean distance
    def euclidean_distance(point1, point2):
        return np.sqrt((point1[0] - point2[0])**2 + (point1[1] - point2[1])**2)

    # Find the gene closest to the bottom right
    scatter_df['distance_to_bottom_right'] = scatter_df.apply(lambda row: euclidean_distance((row['mean_corr_with_A'], row['mean_corr_with_B']), bottom_right_point), axis=1)
    bottom_right_gene = scatter_df.loc[scatter_df['distance_to_bottom_right'].idxmin()]

    # Find the gene closest to the top left
    scatter_df['distance_to_top_left'] = scatter_df.apply(lambda row: euclidean_distance((row['mean_corr_with_A'], row['mean_corr_with_B']), top_left_point), axis=1)
    top_left_gene = scatter_df.loc[scatter_df['distance_to_top_left'].idxmin()]

    # Add labels to the plot
    ax.text(bottom_right_gene['mean_corr_with_A'], bottom_right_gene['mean_corr_with_B'], bottom_right_gene['gene'], color='blue', ha='right')
    ax.text(top_left_gene['mean_corr_with_A'], top_left_gene['mean_corr_with_B'], top_left_gene['gene'], color='red', ha='left')

# Assuming gene_set_A and gene_set_B are lists of gene names
# Assuming adata_gex is your AnnData object containing scRNAseq data
gene_indices = {gene: adata_gex.var.index.get_loc(gene) for gene in set(gene_set_A + gene_set_B)}

# Get all unique categories from 'pseudotime_bin' and move 'post_bifurcation' to the end
bin_categories = [cat for cat in adata_gex.obs['pseudotime_bin'].cat.categories if cat != 'post_bifurcation']
bin_categories.append('post_bifurcation')

# Number of bins to create subplots for
num_bins = len(adata_gex.obs['pseudotime_bin'].cat.categories)

# Create a figure with two rows of subplots: one for UMAP and one for scatter plots
fig, axes = plt.subplots(2, num_bins, figsize=(6 * num_bins, 12))

# Loop over each pseudotime bin to create UMAP and scatter plots
for i, bin_label in enumerate(adata_gex.obs['pseudotime_bin'].cat.categories):
    # Skip the post_bifurcation bin if needed
    #if bin_label == 'post_bifurcation':
    #    continue

    # Filter cells in the current bin for scatter plot
    bin_cells = adata_gex[adata_gex.obs['pseudotime_bin'] == bin_label]

    # Calculate scatter plot data
    scatter_df = calculate_correlation(bin_cells, gene_set_A, gene_set_B, gene_indices)

    # Create UMAP plot in the first row
    ax_umap = axes[0, i]
    sc.pl.umap(adata_gex, color="pseudotime_bin", groups=[bin_label], ax=ax_umap, show=False, legend_loc=None, title="", size=50)

    # Create specific scatter plot in the second row
    ax_scatter = axes[1, i]
    sns.scatterplot(data=scatter_df, x='mean_corr_with_A', y='mean_corr_with_B', hue='gene_set', ax=ax_scatter)
    ax_scatter.set_title(f'{bin_label}')
    ax_scatter.axhline(0, color='grey', linestyle='--')
    ax_scatter.axvline(0, color='grey', linestyle='--')
    ax_scatter.set_xlabel('Correlation with Left aCM early module')
    ax_scatter.set_ylabel('Correlation with Right aCM early module')
    ax_scatter.set_xlim(-1, 1)
    ax_scatter.set_ylim(-1, 1)
    ax_scatter.legend(title='Gene Module')

    # Label the extreme genes
    label_extreme_genes(scatter_df, ax_scatter)

plt.tight_layout()
plt.show()
```

Example 160 (unknown):
```unknown
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 18.84it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                                                                                                    | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 20.25it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                                                                                                    | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 21.55it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                                                                                                    | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 21.89it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:48<00:00,  2.20s/it]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 161 (unknown):
```unknown
early_modules_dict={}
early_modules_dict["LEFT"]=gene_set_A
early_modules_dict["RIGHT"]=gene_set_B
```

Example 162 (python):
```python
import pickle

key="aCM_n570" # or (aCM_n1140 or aCM_full)
key="aCM_n1140"
key="aCM_full"
target_type="Gene"
# File path
file_path = f"/nfs/team205/heart/pyscenic/plus/eRegulon_signatures/{key}_{target_type}_eRegulon_signatures.pkl"

# Open the file and load the dictionary
with open(file_path, 'rb') as file:
    TF_TG_dict = pickle.load(file)
```

Example 163 (unknown):
```unknown
results_list=[]
```

Example 164 (python):
```python
side="LEFT"
associated_features_dict.keys()
total_transcriptome_genes=36000
genes_of_interest=early_modules_dict[f"{side}"]

import pandas as pd
from scipy.stats import fisher_exact


# Initialize a list to store the results
results = []

for regulon, regulon_genes in TF_TG_dict.items():
    # Contingency table
    overlap = len(set(regulon_genes) & set(genes_of_interest))
    only_genes_of_interest = len(genes_of_interest) - overlap
    only_regulon_genes = len(regulon_genes) - overlap
    neither = total_transcriptome_genes - (overlap + only_genes_of_interest + only_regulon_genes)
    table = [[overlap, only_genes_of_interest], [only_regulon_genes, neither]]

    # Perform Fisher's Exact Test
    odds_ratio, p_value = fisher_exact(table)
    
    # Append the results
    results.append({"Regulon": regulon, "Odds Ratio": odds_ratio, "pval": p_value})

# Convert the list of dictionaries to a DataFrame
results_df = pd.DataFrame(results)

# Sort the DataFrame by p-value
results_df.sort_values(by="pval", inplace=True)

# Reset index after sorting
results_df.reset_index(drop=True, inplace=True)

# Display the DataFrame
results_df=results_df[results_df.pval<0.01]
results_df["side"]=side
results_df=results_df.reset_index(names='rank')
results_list.append(results_df)
results_df
```

Example 165 (python):
```python
side="RIGHT"
associated_features_dict.keys()
total_transcriptome_genes=36000
genes_of_interest=early_modules_dict[f"{side}"]

import pandas as pd
from scipy.stats import fisher_exact


# Initialize a list to store the results
results = []

for regulon, regulon_genes in TF_TG_dict.items():
    # Contingency table
    overlap = len(set(regulon_genes) & set(genes_of_interest))
    only_genes_of_interest = len(genes_of_interest) - overlap
    only_regulon_genes = len(regulon_genes) - overlap
    neither = total_transcriptome_genes - (overlap + only_genes_of_interest + only_regulon_genes)
    table = [[overlap, only_genes_of_interest], [only_regulon_genes, neither]]

    # Perform Fisher's Exact Test
    odds_ratio, p_value = fisher_exact(table)
    
    # Append the results
    results.append({"Regulon": regulon, "Odds Ratio": odds_ratio, "pval": p_value})

# Convert the list of dictionaries to a DataFrame
results_df = pd.DataFrame(results)

# Sort the DataFrame by p-value
results_df.sort_values(by="pval", inplace=True)

# Reset index after sorting
results_df.reset_index(drop=True, inplace=True)

# Display the DataFrame
results_df=results_df[results_df.pval<0.01]
results_df["side"]=side
results_df=results_df.reset_index(names='rank')
results_list.append(results_df)
results_df
```

Example 166 (python):
```python
results_concat=pd.concat(results_list)
print(results_concat.shape)
results_concat["TF"]=results_concat.Regulon.str.split("_",expand=True)[0]
results_concat
```

Example 167 (python):
```python
chdgene_table=pd.read_csv("chdgene_table.csv")
results_concat=results_concat.merge(chdgene_table, left_on="TF",right_on="Gene", how='left')
results_concat
```

Example 168 (python):
```python
import matplotlib
matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42
```

Example 169 (python):
```python
# Transforming p-values to -log10(p)
results_concat['-log10(pval)'] = -np.log10(results_concat['pval'])

# Create a pivot table for the heatmap
heatmap_data = results_concat.pivot("TF", "side", "-log10(pval)")
heatmap_data = heatmap_data.fillna(value=0)
heatmap_data = heatmap_data.sort_values(by=["LEFT","RIGHT"], ascending=False)

# Initialize with empty strings
annotations = pd.DataFrame('', index=heatmap_data.index, columns=heatmap_data.columns)

# Loop through the original dataframe to populate the annotations
for _, row in results_concat.iterrows():
    tf = row['TF']
    side = row['side']
    chd_classification = row['CHD classification']
    if pd.notna(chd_classification):
        if "ASD" in chd_classification:
            annotations.loc[tf, side] = '**'  # Two asterisks if "ASD" in CHD classification
        else:
            annotations.loc[tf, side] = '*'   # One asterisk otherwise

# Create the heatmap
plt.figure(figsize=(10, 10))
sns.heatmap(heatmap_data, annot=annotations, fmt="s", cmap="RdPu", square=True, cbar=True)
plt.title("Enrichment of early gene modules for regulons\n '*': associated with any CHD, '**' associated with ASD\ncolor scale = -log10(p-value)")
plt.xlabel("Side")
plt.ylabel("Regulon (Transcription Factor)")
plt.savefig("CHD_enrichment.pdf", format='pdf', bbox_inches='tight')
plt.show()
```

Example 170 (unknown):
```unknown
/tmp/ipykernel_615173/3468513932.py:5: FutureWarning: In a future version of pandas all arguments of DataFrame.pivot will be keyword-only.
  heatmap_data = results_concat.pivot("TF", "side", "-log10(pval)")
```

Example 171 (unknown):
```unknown
[reg for reg in list(TF_TG_dict.keys()) if "PITX" in reg]
```

Example 172 (unknown):
```unknown
['PITX2_+_(130g)']
```

Example 173 (unknown):
```unknown
plot.GO_plot(gene_list=TF_TG_dict["PITX2_+_(130g)"],display_table=True,gene_sets=['GO_Biological_Process_2023'])
```

Example 174 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 175 (python):
```python
sc.pl.dotplot(adata_gex, groupby="AtrialCardiomyocytesImmature_annotation",var_names=TF_TG_dict["PITX2_+_(130g)"])
```

Example 176 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/Visium_mapping/c2l_pt3_plot_FFPE.html

**Contents:**
- Functions¶
- Read in metadata¶
- Data, parameters¶

2045 rows × 5 columns

2045 rows × 8 columns

**Examples:**

Example 1 (unknown):
```unknown
run='20Mar2024_fetal_FFPE'
```

Example 2 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 3 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Global seed set to 0
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 4 (python):
```python
import gc
import torch
```

Example 5 (python):
```python
def read_and_qc(sample_name, path):
    r""" This function reads the data for one 10X spatial experiment into the anndata object.
    It also calculates QC metrics. Modify this function if required by your workflow.

    :param sample_name: Name of the sample
    :param path: path to data
    """

    adata = sc.read_visium(path + str(sample_name),
                           count_file='filtered_feature_bc_matrix.h5', load_images=True)
    adata.var_names_make_unique(join='-')
    
    adata.obs['sample'] = sample_name
    
    # rename genes to ENSEMBL
    adata.var['SYMBOL'] = adata.var_names
    adata.var_names = adata.var['gene_ids']
    adata.var_names.name = None

    # Calculate QC metrics
    sc.pp.calculate_qc_metrics(adata, inplace=True)
    adata.var['mt'] = [gene.startswith('MT-') for gene in adata.var['SYMBOL']]
    adata.obs['mt_frac'] = adata[:, adata.var['mt'].tolist()].X.sum(1).A.squeeze()/adata.obs['total_counts']

    # add sample name to obs names
    adata.obs["sample"] = [str(i) for i in adata.obs['sample']]
    adata.obs_names = adata.obs["sample"] \
                          + '_' + adata.obs_names
    adata.obs.index.name = 'spot_id'

    return adata

def select_slide(adata, s, s_col='sample'):
    r""" This function selects the data for one slide from the spatial anndata object.

    :param adata: Anndata object with multiple spatial experiments
    :param s: name of selected experiment
    :param s_col: column in adata.obs listing experiment name for each location
    """

    slide = adata[adata.obs[s_col].isin([s]), :]
    s_keys = list(slide.uns['spatial'].keys())
    s_spatial = np.array(s_keys)[[s in k for k in s_keys]][0]

    slide.uns['spatial'] = {s_spatial: slide.uns['spatial'][s_spatial]}

    return slide
```

Example 6 (python):
```python
meta=pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230208_for-cell2loc.csv',
                index_col=0)
meta
```

Example 7 (python):
```python
pd.crosstab(meta['donor'],meta['cell2loc-map_group'])
```

Example 8 (unknown):
```unknown
# select slides
# Healthy FFPE
meta = meta[meta['cell2loc-map_group'].isin(['FFPE_16PCW'])]

sampleIDs = list(meta['sangerID'])
len(sampleIDs)
```

Example 9 (python):
```python
import os
results_folder = os.path.join(os.getcwd(),"analysis/Visium/cell2location_results")

ref_run_name = f'{results_folder}/reference_signatures/{run}'
run_name = f'{results_folder}/cell2location_map/{run}'
```

Example 10 (unknown):
```unknown
visium_path='/nfs/team205/heart/mapped/spaceranger200/'

celltype_key='fine_grain_modified'

vis_subset_key='cell2loc-map_group'
```

Example 11 (python):
```python
%%time
subset_values = os.listdir(os.path.join(results_folder, "cell2location_map", run))
print(f"Found these subsets {subset_values}")

for value in subset_values:
    print(f"###### {value} ######")
    adata_sub = sc.read(f"{run_name}/{value}/sp.h5ad")
    print("visium read in")
    adata_sub.obs[adata_sub.uns['mod']['factor_names']] = adata_sub.obsm['q05_cell_abundance_w_sf']
    IDs = meta[meta[vis_subset_key] == value].sangerID.tolist()
    print(f"{value}: {IDs}")

    for sample in IDs:
        try:
            slide = select_slide(adata_sub, sample)
        except IndexError as e:
            print(f"Sample {sample} not found in adata: {e}")
            continue  # Skip this sample if not found

        slide.obs[slide.uns['mod']['factor_names']] = slide.obsm['q05_cell_abundance_w_sf']
        list_of_cell_types = slide.uns['mod']['factor_names'].tolist()
        print("plotting HE image...")
        sc.pl.spatial(slide, color=None, size=0, img_key='hires', title=f"HE:{sample}", 
                      save=f"{run}__{value}__{sample}__HE.png", show=False)

        # Plotting abundances for all relevant cell states
        if list_of_cell_types:
            print("plotting abundances for all cell states...")
            sc.pl.spatial(slide, color=list_of_cell_types, cmap="RdPu", ncols=10, size=1.3, 
                          img_key='hires', vmin=0, vmax='p99.2', alpha_img=0.3, 
                          save=f"{run}__{value}__{sample}__all_cell_states_abundances.png", show=False)
```

Example 12 (unknown):
```unknown
Found these subsets ['5PCW', '20PCW', '13PCW', '4PCW', '7PCW', 'FFPE_16PCW']
###### 5PCW ######
visium read in
5PCW: []
###### 20PCW ######
visium read in
20PCW: []
###### 13PCW ######
visium read in
13PCW: []
###### 4PCW ######
visium read in
4PCW: []
###### 7PCW ######
visium read in
7PCW: []
###### FFPE_16PCW ######
visium read in
FFPE_16PCW: ['BHF_F_Hea13295830', 'BHF_F_Hea13295831']
```

Example 13 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 14 (unknown):
```unknown
plotting HE image...
```

Example 15 (python):
```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File <timed exec>:22

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:1003, in spatial(adata, basis, img, img_key, library_id, crop_coord, alpha_img, bw, size, scale_factor, spot_size, na_color, show, return_fig, save, **kwargs)
   1000     cmap_img = None
   1001 circle_radius = size * scale_factor * spot_size * 0.5
-> 1003 axs = embedding(
   1004     adata,
   1005     basis=basis,
   1006     scale_factor=scale_factor,
   1007     size=circle_radius,
   1008     na_color=na_color,
   1009     show=False,
   1010     save=False,
   1011     **kwargs,
   1012 )
   1013 if not isinstance(axs, list):
   1014     axs = [axs]

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392, in embedding(adata, basis, color, gene_symbols, use_raw, sort_order, edges, edges_width, edges_color, neighbors_key, arrows, arrows_kwds, groups, components, dimensions, layer, projection, scale_factor, color_map, cmap, palette, na_color, na_in_legend, size, frameon, legend_fontsize, legend_fontweight, legend_loc, legend_fontoutline, colorbar_loc, vmax, vmin, vcenter, norm, add_outline, outline_width, outline_color, ncols, hspace, wspace, title, show, save, ax, return_fig, **kwargs)
    389         # if user did not set alpha, set alpha to 0.7
    390         kwargs['alpha'] = 0.7 if alpha is None else alpha
--> 392     cax = scatter(
    393         coords[:, 0],
    394         coords[:, 1],
    395         marker=".",
    396         c=color_vector,
    397         rasterized=settings._vector_friendly,
    398         norm=normalize,
    399         **kwargs,
    400     )
    402 # remove y and x ticks
    403 ax.set_yticks([])

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_utils.py:1108, in circles(x, y, s, ax, marker, c, vmin, vmax, scale_factor, **kwargs)
   1105 # You can set `facecolor` with an array for each patch,
   1106 # while you can only set `facecolors` with a value for all.
   1107 if scale_factor != 1.0:
-> 1108     x = x * scale_factor
   1109     y = y * scale_factor
   1110 zipped = np.broadcast(x, y, s)

TypeError: can't multiply sequence by non-int of type 'float'
```

Example 16 (python):
```python
%%time

adata_dict={}

subset_values = os.listdir(os.path.join(results_folder, "cell2location_map", run))
print(f"Found these subsets {subset_values}")

for value in subset_values:
    print(f"###### {value} ######")
    adata_sub = sc.read(f"{run_name}/{value}/sp.h5ad")
    print("visium read in")
    adata_sub.obs[adata_sub.uns['mod']['factor_names']] = adata_sub.obsm['q05_cell_abundance_w_sf']
    IDs = meta[meta[vis_subset_key] == value].sangerID.tolist()
    print(f"{value}: {IDs}")

    for sample in IDs:
        try:
            slide = select_slide(adata_sub, sample)
        except IndexError as e:
            print(f"Sample {sample} not found in adata: {e}")
            continue  # Skip this sample if not found

        slide.obs[slide.uns['mod']['factor_names']] = slide.obsm['q05_cell_abundance_w_sf']
        adata_dict[sample]=slide
```

Example 17 (unknown):
```unknown
Found these subsets ['5PCW', '20PCW', '13PCW', '4PCW', '7PCW', 'FFPE_16PCW']
###### 5PCW ######
visium read in
5PCW: []
###### 20PCW ######
visium read in
20PCW: []
###### 13PCW ######
visium read in
13PCW: []
###### 4PCW ######
visium read in
4PCW: []
###### 7PCW ######
visium read in
7PCW: []
###### FFPE_16PCW ######
visium read in
FFPE_16PCW: ['BHF_F_Hea13295830', 'BHF_F_Hea13295831']
```

Example 18 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 19 (unknown):
```unknown
CPU times: user 2.11 s, sys: 8.17 s, total: 10.3 s
Wall time: 15.2 s
```

Example 20 (python):
```python
adata_dict.keys()
```

Example 21 (unknown):
```unknown
dict_keys(['BHF_F_Hea13295830', 'BHF_F_Hea13295831'])
```

Example 22 (python):
```python
from tqdm import tqdm
atrial_data_dict={}
for sample,tmp_adata in tqdm(adata_dict.items()):
    print(f"sample: {sample},  age is {tmp_adata.obs.age.unique().tolist()[0]}")
    tmp_df=tmp_adata[tmp_adata.obs.JC_edit=="Myocardium - Atrial"].obs[['AtrialCardiomyocytesImmature', 'AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight']]

    if tmp_df.shape[0]>0:
        tmp_df["sample"]=sample
        tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
        atrial_data_dict[sample]=tmp_df
    else:
        print(f"no atrial myocardium spots annotated for sample: {sample}")
```

Example 23 (python):
```python
0%|                                                                                                                                                                                                                   | 0/2 [00:00<?, ?it/s]/tmp/ipykernel_1570535/2563514209.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_1570535/2563514209.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_1570535/2563514209.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_1570535/2563514209.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 89.72it/s]
```

Example 24 (unknown):
```unknown
sample: BHF_F_Hea13295830,  age is PCW-16w
sample: BHF_F_Hea13295831,  age is PCW-16w
```

Example 25 (python):
```python
data=pd.concat(list(atrial_data_dict.values()))
data
```

Example 26 (unknown):
```unknown
# Assuming 'data' is your pandas DataFrame
# Calculate the total for each row
total = data['AtrialCardiomyocytesImmature'] + data['AtrialCardiomyocytesRight'] + data['AtrialCardiomyocytesLeft']

# Calculate the proportions for each column
data['AtrialCardiomyocytesImmature_prop'] = data['AtrialCardiomyocytesImmature'] / total
data['AtrialCardiomyocytesRight_prop'] = data['AtrialCardiomyocytesRight'] / total
data['AtrialCardiomyocytesLeft_prop'] = data['AtrialCardiomyocytesLeft'] / total

# The new DataFrame 'data' now includes the proportion columns
data
```

Example 27 (python):
```python
import plotly.express as px
fig = px.scatter_ternary(data,
                         a="AtrialCardiomyocytesImmature_prop", b="AtrialCardiomyocytesLeft_prop", c="AtrialCardiomyocytesRight_prop",
                        color="age",
                        opacity=0.8)
fig.show()
```

Example 28 (unknown):
```unknown
data.to_csv("AtrialProportionsFFPE.csv")
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/no1_compact-vs-trab_region-signatures.html

**Contents:**
- Function¶
- Read in adata¶
- ~Read in DAR result: compact-vs-trab~¶
- Read in fitted result, gene¶
- Select fitted genes which were highest in the group_a or group_b¶
- Fitted peaks¶
- Save the peaks¶
- Scoring and plot¶

i.e. separating into UP or DOWN regulated genes

4483 rows × 6 columns

**Examples:**

Example 1 (python):
```python
# Parameters
working_dir = "/nfs/team205/heart/Vitro/TF_selection"
path_to_adata_dir = "/nfs/team205/heart/anndata_objects/Vitro/post_multiVI_full-features"
path_to_adata_peak_tfidf = "/nfs/team205/heart/anndata_objects/Foetal/multiome_ATAC/ArchR/project_output/PeakMatrix/Foetal_Peaks_tfidf.h5ad"
group_key = "t_binned"
group_a = "t3"
group_b = "t1"
time_key = "t"
subset_object = False
geneset_list = ["MSigDB_Hallmark_2020", "KEGG_2021_Human", "GO_Biological_Process_2023"]

path_to_eRegulon_signatures_gene = "/nfs/team205/heart/pyscenic/plus/eRegulon_signatures/vCM_onlyLeft_Gene_eRegulon_signatures.pkl"
path_to_eRegulon_signatures_region = "/nfs/team205/heart/pyscenic/plus/eRegulon_signatures/vCM_onlyLeft_Region_eRegulon_signatures.pkl"

path_to_DAR_compact = '/nfs/team205/heart/anndata_objects/Foetal/multiome_ATAC/ArchR/project_output/MarkerPeaks/vCM_Left_Compact-vs-Trab.csv'
path_to_DAR_trab = '/nfs/team205/heart/anndata_objects/Foetal/multiome_ATAC/ArchR/project_output/MarkerPeaks/vCM_Left_Trab-vs-Compact.csv'
path_to_eRegulon_metadata = "/nfs/team205/heart/pyscenic/plus/eRegulon_metadata/vCM_onlyLeft_eRegulon_metadata_filtered.csv" # filtered metadata
```

Example 2 (python):
```python
import scanpy as sc
import pandas as pd
import numpy as np
import anndata

import matplotlib.pyplot as plt
import seaborn as sns
```

Example 3 (python):
```python
import session_info
session_info.show()
```

Example 4 (python):
```python
-----
anndata             0.9.2
matplotlib          3.5.2
numpy               1.22.3
pandas              1.5.3
scanpy              1.8.2
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 5 (python):
```python
PIL                 10.0.0
asttokens           NA
backcall            0.2.0
beta_ufunc          NA
binom_ufunc         NA
cffi                1.15.1
colorama            0.4.6
comm                0.1.4
cpuinfo             NA
cycler              0.10.0
cython_runtime      NA
dateutil            2.8.2
debugpy             1.6.7.post1
decorator           5.1.1
executing           1.2.0
h5py                3.9.0
hypergeom_ufunc     NA
igraph              0.10.6
importlib_metadata  NA
ipykernel           6.25.1
jedi                0.19.0
joblib              1.3.2
kiwisolver          1.4.5
leidenalg           0.10.1
llvmlite            0.40.1
mpl_toolkits        NA
natsort             8.4.0
nbinom_ufunc        NA
numba               0.57.1
numexpr             2.8.5
packaging           23.1
parso               0.8.3
patsy               0.5.3
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
platformdirs        3.10.0
plotly              5.16.1
prompt_toolkit      3.0.39
psutil              5.9.5
ptyprocess          0.7.0
pure_eval           0.2.2
pydev_ipython       NA
pydevconsole        NA
pydevd              2.9.5
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.16.1
pyparsing           3.0.9
pytz                2023.3
scipy               1.8.1
sinfo               0.3.4
six                 1.16.0
sklearn             1.3.0
sphinxcontrib       NA
stack_data          0.6.2
statsmodels         0.14.0
tables              3.8.0
texttable           1.6.7
threadpoolctl       3.2.0
tornado             6.3.3
traitlets           5.9.0
typing_extensions   NA
wcwidth             0.2.6
zipp                NA
zmq                 25.1.1
```

Example 6 (unknown):
```unknown
-----
IPython             8.12.2
jupyter_client      8.3.0
jupyter_core        5.3.1
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-4.15.0-212-generic-x86_64-with-glibc2.17
-----
Session information updated at 2023-10-05 08:46
```

Example 7 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 8 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 9 (python):
```python
# read in normalised data
adata_tfidf = sc.read_h5ad(path_to_adata_peak_tfidf)
print(adata_tfidf.X.data[:10])
adata_tfidf
```

Example 10 (unknown):
```unknown
[1.768611  1.9229858 3.635446  1.3192275 1.4049321 1.8123279 1.6739427
 2.9703567 3.3852873 1.2617257]
```

Example 11 (unknown):
```unknown
AnnData object with n_obs × n_vars = 167022 × 508040
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels'
    var: 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
```

Example 12 (python):
```python
# read in compact, trabeculated, all adata
# which has pseudotime, etc

print('### compact ###')
adata_compact = sc.read_h5ad(f'{path_to_adata_dir}/Foetal_vCM_Left-compact_atac.h5ad')
# filter features to remove those that appear in fewer than 0.5% of the cells
print(adata_compact.shape)
sc.pp.filter_genes(adata_compact, min_cells=int(adata_compact.shape[0] * 0.005))
print(adata_compact.shape)
# replace .X data with tf-idf normalised data
adata_compact.X = adata_tfidf[adata_compact.obs_names,adata_compact.var_names].X
print(adata_compact.X.data[:10])
print(adata_compact)

print('### trabeculated ###')
adata_trab = sc.read_h5ad(f'{path_to_adata_dir}/Foetal_vCM_Left-trabeculated_atac.h5ad')
# filter features to remove those that appear in fewer than 0.5% of the cells
print(adata_trab.shape)
sc.pp.filter_genes(adata_trab, min_cells=int(adata_trab.shape[0] * 0.005))
print(adata_trab.shape)
# replace .X data with tf-idf normalised data
adata_trab.X = adata_tfidf[adata_trab.obs_names,adata_trab.var_names].X
print(adata_trab.X.data[:10])
print(adata_trab)

'''
print('### all ###')
adata_all = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.h5ad')
# subset peak feature data
adata_all = adata_all[:,adata_all.var['modality']=='Peaks']
# select features
# features = list(set(list(adata_compact.var_names)+list(adata_trab.var_names)))
# adata_all = adata_all[:,features]
# replace .X data with tf-idf normalised data
adata_all.X = adata_tfidf[adata_all.obs_names,adata_all.var_names].X
print(adata_all.X.data[:10])
print(adata_all)
'''
```

Example 13 (unknown):
```unknown
### compact ###
(11397, 508040)
(11397, 242879)
[1.885513  4.3636436 1.0840797 2.524973  1.4176022 2.0811439 2.3172326
 2.7016861 0.8455251 4.038132 ]
AnnData object with n_obs × n_vars = 11397 × 242879
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont', 't_binned'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N', 'n_cells'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 't_binned_colors', 'umap', 'week_colors'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
### trabeculated ###
(5054, 508040)
(5054, 258442)
[3.2284448 4.816777  4.724808  3.9381142 2.5444632 4.71782   3.8803818
 4.8070517 3.4701438 5.037634 ]
AnnData object with n_obs × n_vars = 5054 × 258442
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont', 't_binned'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N', 'n_cells'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 't_binned_colors', 'umap', 'week_colors'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
```

Example 14 (python):
```python
"\nprint('### all ###')\nadata_all = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.h5ad')\n#\xa0subset peak feature data\nadata_all = adata_all[:,adata_all.var['modality']=='Peaks']\n#\xa0select features\n#\xa0features = list(set(list(adata_compact.var_names)+list(adata_trab.var_names)))\n#\xa0adata_all = adata_all[:,features]\n#\xa0replace .X data with tf-idf normalised data\nadata_all.X = adata_tfidf[adata_all.obs_names,adata_all.var_names].X\nprint(adata_all.X.data[:10])\nprint(adata_all)\n"
```

Example 15 (python):
```python
sc.pl.draw_graph(adata_compact, color=[group_key, time_key], size=5, wspace=0.3, ncols=3)
```

Example 16 (python):
```python
sc.pl.draw_graph(adata_trab, color=[group_key, time_key], size=5, wspace=0.3, ncols=3)
```

Example 17 (python):
```python
'''
logFC_threshold = 0.1
FDR_threshold = 1

# compact
compact_dar = pd.read_csv(path_to_DAR_compact)
compact_dar_up = compact_dar[(compact_dar['Log2FC']> logFC_threshold)&(compact_dar['FDR']<FDR_threshold)]
compact_dar_down = compact_dar[(compact_dar['Log2FC']< -logFC_threshold)&(compact_dar['FDR']<FDR_threshold)]
print(f'compact, up: {len(compact_dar_up)} regions')
print(f'compact, down: {len(compact_dar_down)} regions')
'''
```

Example 18 (python):
```python
fit_res_compact = pd.read_csv(f'{working_dir}/foetal_vCM_left-compact/outs/Fitting_{time_key}_Peak.csv',index_col=0)
fit_res_trab = pd.read_csv(f'{working_dir}/foetal_vCM_left-trabeculated/outs/Fitting_{time_key}_Peak.csv',index_col=0)
```

Example 19 (unknown):
```unknown
fit_res_compact.head()
```

Example 20 (unknown):
```unknown
fit_res_compact.loc['CDKN1A']
```

Example 21 (python):
```python
# compact
group_mean_compact = utils.grouped_obs_mean(adata_compact,group_key=group_key)

# select upregulated (a>b) or downregulated (b>a) genes
group_a_peaks_compact = list(group_mean_compact.index[group_mean_compact[group_a] > group_mean_compact[group_b]])
group_b_peaks_compact = list(group_mean_compact.index[group_mean_compact[group_b] > group_mean_compact[group_a]])

group_mean_compact.head()
```

Example 22 (python):
```python
# trabeculated
group_mean_trab = utils.grouped_obs_mean(adata_trab,group_key=group_key)

# select upregulated (a>b) or downregulated (b>a) genes
group_a_peaks_trab = list(group_mean_trab.index[group_mean_trab[group_a] > group_mean_trab[group_b]])
group_b_peaks_trab = list(group_mean_trab.index[group_mean_trab[group_b] > group_mean_trab[group_a]])

group_mean_trab.head()
```

Example 23 (unknown):
```unknown
fit_fdr_threshold = 0.05
fit_A_threshold = 0.5
```

Example 24 (python):
```python
# compact
fit_res_sel_compact = fit_res_compact[(fit_res_compact['fdr']<fit_fdr_threshold)&\
                                      (fit_res_compact['A']>fit_A_threshold)
                                     ]
# group a, upregulated
shared = list(set(fit_res_sel_compact.index).intersection(group_a_peaks_compact))
fit_res_sel_up2a_compact = fit_res_sel_compact.loc[shared]
fit_res_sel_up2a_compact = fit_res_sel_up2a_compact.sort_values('A',ascending=False)
print(f'up towards {group_a}, number of fitted peaks: {len(fit_res_sel_up2a_compact)}')
# group b, downregulated
shared = list(set(fit_res_sel_compact.index).intersection(group_b_peaks_compact))
fit_res_sel_up2b_compact = fit_res_sel_compact.loc[shared]
fit_res_sel_up2b_compact = fit_res_sel_up2b_compact.sort_values('A',ascending=False)
print(f'up towards {group_b}, number of fitted peaks: {len(fit_res_sel_up2b_compact)}')
```

Example 25 (unknown):
```unknown
up towards t3, number of fitted peaks: 4483
up towards t1, number of fitted peaks: 14499
```

Example 26 (python):
```python
# trabeculated
fit_res_sel_trab = fit_res_trab[(fit_res_trab['fdr']<fit_fdr_threshold)&\
                                (fit_res_trab['A']>fit_A_threshold)
                                     ]
# group a, upregulated
shared = list(set(fit_res_sel_trab.index).intersection(group_a_peaks_trab))
fit_res_sel_up2a_trab = fit_res_sel_trab.loc[shared]
fit_res_sel_up2a_trab = fit_res_sel_up2a_trab.sort_values('A',ascending=False)
print(f'up towards {group_a}, number of fitted peaks: {len(fit_res_sel_up2a_trab)}')
# group b, downregulated
shared = list(set(fit_res_sel_trab.index).intersection(group_b_peaks_trab))
fit_res_sel_up2b_trab = fit_res_sel_trab.loc[shared]
fit_res_sel_up2b_trab = fit_res_sel_up2b_trab.sort_values('A',ascending=False)
print(f'up towards {group_b}, number of fitted peaks: {len(fit_res_sel_up2b_trab)}')
```

Example 27 (unknown):
```unknown
up towards t3, number of fitted peaks: 4147
up towards t1, number of fitted peaks: 21577
```

Example 28 (python):
```python
sns.scatterplot(fit_res_compact,
                x='fdr',
                y='A',
                size=0.001
                )
```

Example 29 (unknown):
```unknown
<AxesSubplot:xlabel='fdr', ylabel='A'>
```

Example 30 (python):
```python
sns.scatterplot(fit_res_trab,
                x='fdr',
                y='A',
                size=0.001
                )
```

Example 31 (unknown):
```unknown
<AxesSubplot:xlabel='fdr', ylabel='A'>
```

Example 32 (python):
```python
?sns.scatterplot
```

Example 33 (python):
```python
Signature:
sns.scatterplot(
    data=None,
    *,
    x=None,
    y=None,
    hue=None,
    size=None,
    style=None,
    palette=None,
    hue_order=None,
    hue_norm=None,
    sizes=None,
    size_order=None,
    size_norm=None,
    markers=True,
    style_order=None,
    legend='auto',
    ax=None,
    **kwargs,
)
Docstring:
Draw a scatter plot with possibility of several semantic groupings.

The relationship between `x` and `y` can be shown for different subsets
of the data using the `hue`, `size`, and `style` parameters. These
parameters control what visual semantics are used to identify the different
subsets. It is possible to show up to three dimensions independently by
using all three semantic types, but this style of plot can be hard to
interpret and is often ineffective. Using redundant semantics (i.e. both
`hue` and `style` for the same variable) can be helpful for making
graphics more accessible.

See the :ref:`tutorial <relational_tutorial>` for more information.

The default treatment of the `hue` (and to a lesser extent, `size`)
semantic, if present, depends on whether the variable is inferred to
represent "numeric" or "categorical" data. In particular, numeric variables
are represented with a sequential colormap by default, and the legend
entries show regular "ticks" with values that may or may not exist in the
data. This behavior can be controlled through various parameters, as
described and illustrated below.

Parameters
----------
data : :class:`pandas.DataFrame`, :class:`numpy.ndarray`, mapping, or sequence
    Input data structure. Either a long-form collection of vectors that can be
    assigned to named variables or a wide-form dataset that will be internally
    reshaped.
x, y : vectors or keys in ``data``
    Variables that specify positions on the x and y axes.
hue : vector or key in `data`
    Grouping variable that will produce points with different colors.
    Can be either categorical or numeric, although color mapping will
    behave differently in latter case.
size : vector or key in `data`
    Grouping variable that will produce points with different sizes.
    Can be either categorical or numeric, although size mapping will
    behave differently in latter case.
style : vector or key in `data`
    Grouping variable that will produce points with different markers.
    Can have a numeric dtype but will always be treated as categorical.
palette : string, list, dict, or :class:`matplotlib.colors.Colormap`
    Method for choosing the colors to use when mapping the ``hue`` semantic.
    String values are passed to :func:`color_palette`. List or dict values
    imply categorical mapping, while a colormap object implies numeric mapping.
hue_order : vector of strings
    Specify the order of processing and plotting for categorical levels of the
    ``hue`` semantic.
hue_norm : tuple or :class:`matplotlib.colors.Normalize`
    Either a pair of values that set the normalization range in data units
    or an object that will map from data units into a [0, 1] interval. Usage
    implies numeric mapping.
sizes : list, dict, or tuple
    An object that determines how sizes are chosen when `size` is used.
    List or dict arguments should provide a size for each unique data value,
    which forces a categorical interpretation. The argument may also be a
    min, max tuple.
size_order : list
    Specified order for appearance of the `size` variable levels,
    otherwise they are determined from the data. Not relevant when the
    `size` variable is numeric.
size_norm : tuple or Normalize object
    Normalization in data units for scaling plot objects when the
    `size` variable is numeric.
markers : boolean, list, or dictionary
    Object determining how to draw the markers for different levels of the
    `style` variable. Setting to `True` will use default markers, or
    you can pass a list of markers or a dictionary mapping levels of the
    `style` variable to markers. Setting to `False` will draw
    marker-less lines.  Markers are specified as in matplotlib.
style_order : list
    Specified order for appearance of the `style` variable levels
    otherwise they are determined from the data. Not relevant when the
    `style` variable is numeric.
legend : "auto", "brief", "full", or False
    How to draw the legend. If "brief", numeric `hue` and `size`
    variables will be represented with a sample of evenly spaced values.
    If "full", every group will get an entry in the legend. If "auto",
    choose between brief or full representation based on number of levels.
    If `False`, no legend data is added and no legend is drawn.
ax : :class:`matplotlib.axes.Axes`
    Pre-existing axes for the plot. Otherwise, call :func:`matplotlib.pyplot.gca`
    internally.
kwargs : key, value mappings
    Other keyword arguments are passed down to
    :meth:`matplotlib.axes.Axes.scatter`.

Returns
-------
:class:`matplotlib.axes.Axes`
    The matplotlib axes containing the plot.

See Also
--------
lineplot : Plot data using lines.
stripplot : Plot a categorical scatter with jitter.
swarmplot : Plot a categorical scatter with non-overlapping points.

Examples
--------

.. include:: ../docstrings/scatterplot.rst
File:      ~/my-conda-envs/vitro/lib/python3.8/site-packages/seaborn/relational.py
Type:      function
```

Example 34 (python):
```python
peak_signatures = {}
# up, towards group_a
peak_signatures['compact_up'] = list(fit_res_sel_up2a_compact.index)
peak_signatures['trab_up'] = list(fit_res_sel_up2a_trab.index)

# down, towards group_b
peak_signatures['compact_down'] = list(fit_res_sel_up2b_compact.index)
peak_signatures['trab_down'] = list(fit_res_sel_up2b_trab.index)

print(peak_signatures.keys())

# save
utils.save_pkl(peak_signatures,
           file = f'{working_dir}/foetal_vCM_left_compact-vs-trabeculated_downstream/peak_signatures.pkl'
          )
```

Example 35 (unknown):
```unknown
dict_keys(['compact_up', 'trab_up', 'compact_down', 'trab_down'])
```

Example 36 (unknown):
```unknown
f'{working_dir}/foetal_vCM_left_compact-vs-trabeculated_downstream/peak_signatures.pkl'
```

Example 37 (unknown):
```unknown
'/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/peak_signatures.pkl'
```

Example 38 (python):
```python
adata_compact.X.data[:10]
```

Example 39 (unknown):
```unknown
array([1.885513 , 4.3636436, 1.0840797, 2.524973 , 1.4176022, 2.0811439,
       2.3172326, 2.7016861, 0.8455251, 4.038132 ], dtype=float32)
```

Example 40 (unknown):
```unknown
fit_res_sel_up2a_compact
```

Example 41 (python):
```python
# compact
sc.tl.score_genes(adata_compact,peak_signatures['compact_up'],score_name=f'compact-up ({len(peak_signatures["compact_up"])}r)')
sc.tl.score_genes(adata_compact,peak_signatures['compact_down'],score_name=f'compact-down ({len(peak_signatures["compact_down"])}r)')

sc.pl.draw_graph(adata_compact, color=['fine_grain',
                                   f'compact-up ({len(peak_signatures["compact_up"])}r)',
                                   f'compact-down ({len(peak_signatures["compact_down"])}r)',
                                  ], 
                 frameon=False, legend_loc='on data',legend_fontsize=0,
                 size=7, wspace=0.1, ncols=4, cmap='YlOrRd')
```

Example 42 (python):
```python
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/tools/_score_genes.py:151: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
  for cut in np.unique(obs_cut.loc[gene_list]):
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/tools/_score_genes.py:151: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
  for cut in np.unique(obs_cut.loc[gene_list]):
```

Example 43 (python):
```python
# trabeculated
sc.tl.score_genes(adata_trab,peak_signatures['trab_up'],score_name=f'trab-up ({len(peak_signatures["trab_up"])}r)')
sc.tl.score_genes(adata_trab,peak_signatures['trab_down'],score_name=f'trab-down ({len(peak_signatures["trab_down"])}r)')

sc.pl.draw_graph(adata_trab, color=['fine_grain',
                                   f'trab-up ({len(peak_signatures["trab_up"])}r)',
                                   f'trab-down ({len(peak_signatures["trab_down"])}r)',
                                  ], 
                 frameon=False, legend_loc='on data',legend_fontsize=0,
                 size=7, wspace=0.1, ncols=4, cmap='YlOrRd')
```

Example 44 (python):
```python
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/tools/_score_genes.py:151: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
  for cut in np.unique(obs_cut.loc[gene_list]):
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/tools/_score_genes.py:151: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
  for cut in np.unique(obs_cut.loc[gene_list]):
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/CellOracle/CellOracle_no2_in-silico-perturbation_AtrialCM-all_out_12May2025.html

**Contents:**
- Import libraries¶
- Make out directory¶
- Load oracle object¶
- In silico TF perturbation analysis¶
- Set parameters¶
- Calculating pseudotime gradient vectors¶
- Perturbation¶

ref: https://morris-lab.github.io/CellOracle.documentation/notebooks/04_Network_analysis/Network_analysis_with_Paul_etal_2015_data.html

**Examples:**

Example 1 (unknown):
```unknown
# Parameters
# working_dir = "/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/CO_in-silico-perturbation_papermill"
path_to_oracle_object = "/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM.celloracle.oracle"
path_to_oracle_links = "/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM_links.celloracle.links"

# path_to_TFs = 

cell_group_name = "AtrialCM"

path_to_res_dict = f"/nfs/team205/heart/anndata_objects/Foetal/celloracle/{cell_group_name}_in-silico_perturbation_May2025.pkl"
path_to_plots = f"/nfs/team205/heart/anndata_objects/Foetal/celloracle/figures_pm/{cell_group_name}_May2025"
```

Example 2 (python):
```python
import os
import sys

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scanpy as sc
import seaborn as sns
```

Example 3 (python):
```python
import celloracle as co
co.__version__
```

Example 4 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 5 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 6 (python):
```python
if not os.path.exists(path_to_plots):
    # Create the directory
    os.makedirs(path_to_plots)
    print("Directory created successfully")
else:
    print("Directory already exists")
```

Example 7 (unknown):
```unknown
Directory already exists
```

Example 8 (unknown):
```unknown
# Load
oracle = co.load_hdf5(path_to_oracle_object)
```

Example 9 (python):
```python
n_cell = oracle.adata.shape[0]
print(f"cell number is :{n_cell}")

k = int(0.025*n_cell)
print(f"Auto-selected k is :{k}")
```

Example 10 (unknown):
```unknown
cell number is :5664
Auto-selected k is :141
```

Example 11 (python):
```python
%matplotlib inline
# Check clustering data
sc.pl.umap(oracle.adata, color='fine_grain_time-binned')
```

Example 12 (python):
```python
# read in post-g2g-selection csv file and get selected TFs
# log2FC<0 means TFs specific to Right-aCM
# log2FC>0 means TFs specific to Left-aCM
g2g_sel = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/AtrialCM_celloracle/AtrialCM_post-g2g_selected-recgulons.csv')
display(g2g_sel)

# TFs, only activator
list_of_TFs = [x.split('_')[0] for x in g2g_sel['Gene'] if '_+' in x]
print(len(list_of_TFs))
```

Example 13 (python):
```python
# select TFs which are in the oracle object
list_of_TFs = [x for x in list_of_TFs if x in oracle.adata.var_names]

print(len(list_of_TFs))
list_of_TFs[:5]
```

Example 14 (unknown):
```unknown
['ETS2', 'MAF', 'IKZF2', 'ETV6', 'BHLHE41']
```

Example 15 (python):
```python
# Check gene expressions
# will be saved to a separated file
sc.settings.set_figure_params(dpi=80)
sc.pl.umap(oracle.adata, color=list_of_TFs, ncols=6,
                 layer="imputed_count", use_raw=False, cmap="RdPu",
                 # show=False, save='_TF_expression.png'
                )
```

Example 16 (unknown):
```unknown
n_grid = 40
n_neighbors = 100
min_mass = 10
scale_simulation = 5
```

Example 17 (python):
```python
from celloracle.applications import Gradient_calculator
# Instantiate Gradient calculator object
gradient = Gradient_calculator(oracle_object=oracle, pseudotime_key="t")
gradient.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=n_neighbors)
gradient.calculate_mass_filter(min_mass=min_mass, plot=True)

# convert the pseudotime data into grid points.
# For this calculation we can chose one of two methods.
gradient.transfer_data_into_grid(args={"method": "knn", "n_knn":50})

# Calculate graddient
gradient.calculate_gradient()
# Show results
scale_dev = 40
gradient.visualize_results(scale=scale_dev, s=5)
# Visualize results
fig, ax = plt.subplots(figsize=[6, 6])
gradient.plot_dev_flow_on_grid(scale=scale_dev, ax=ax)
```

Example 18 (python):
```python
from tqdm import tqdm
```

Example 19 (python):
```python
from celloracle.applications import Oracle_development_module

# not showing plots
matplotlib.use('Agg')

dev_dict = {}
for goi in tqdm(list_of_TFs):
    # Enter perturbation conditions to simulate signal propagation after the perturbation.
    try:
        oracle.simulate_shift(perturb_condition={goi: 0.0}, # this is KO simulation so expression=0
                          n_propagation=3)
        print(f'simulating {goi}')
    except:
        print(f'skipping {goi} due to an error')
        continue
    
    # Get transition probability
    oracle.estimate_transition_prob(n_neighbors=n_neighbors,
                                    knn_random=True,
                                    sampled_fraction=1)

    # Calculate embedding
    oracle.calculate_embedding_shift(sigma_corr=0.05)
    
    # prepare grid
    oracle.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=n_neighbors)
    oracle.calculate_mass_filter(min_mass=min_mass, plot=False)
    
    # plot vector field
    fig, ax = plt.subplots(1, 2,  figsize=[13, 6])
    # Show quiver plot
    oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax[0])
    ax[0].set_title(f"Simulated cell identity shift vector: {goi} KO")
    # Show quiver plot that was calculated with randomized graph.
    oracle.plot_simulation_flow_random_on_grid(scale=scale_simulation, ax=ax[1])
    ax[1].set_title(f"Randomized simulation vector")
    plt.savefig(f'{path_to_plots}/{goi}_simulation_vector.pdf')
    
    # Plot vector field with cell cluster
    fig, ax = plt.subplots(figsize=[8, 8])
    oracle.plot_cluster_whole(ax=ax, s=10)
    oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax, show_background=False)
    plt.savefig(f'{path_to_plots}/{goi}_simulation_vector_with-clusters.pdf')
    
    
    dev = Oracle_development_module()
    # Load development flow
    dev.load_differentiation_reference_data(gradient_object=gradient)

    # Load simulation result
    dev.load_perturb_simulation_data(oracle_object=oracle,
                                     cell_idx_use=None, # Enter cell id list
                                     name=cell_group_name # Name of this cell group. You can enter any name.
                                     )

    # Calculation
    dev.calculate_inner_product()
    dev.calculate_digitized_ip(n_bins=10)
    
    # save to the dictionary
    dev_dict[goi] = dev.copy()
    
    # Let's visualize the results
    vm = 0.01
    dev.visualize_development_module_layout_0(s=5,
                                              scale_for_simulation=scale_simulation,
                                              s_grid=50,
                                              scale_for_pseudotime=scale_dev,
                                              vm=vm)
    plt.savefig(f'{path_to_plots}/{goi}_results.pdf')
```

Example 20 (unknown):
```unknown
0%|          | 0/9 [00:00<?, ?it/s]
```

Example 21 (unknown):
```unknown
simulating ETS2
```

Example 22 (unknown):
```unknown
2025-05-12 11:57:36,089 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-12 11:57:36,112 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
 11%|█         | 1/9 [05:46<46:14, 346.82s/it]
```

Example 23 (unknown):
```unknown
simulating MAF
```

Example 24 (unknown):
```unknown
2025-05-12 12:03:27,557 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-12 12:03:27,574 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
 22%|██▏       | 2/9 [11:38<40:46, 349.51s/it]
```

Example 25 (unknown):
```unknown
simulating IKZF2
```

Example 26 (unknown):
```unknown
2025-05-12 12:09:36,927 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-12 12:09:36,939 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
 33%|███▎      | 3/9 [17:47<35:51, 358.54s/it]
```

Example 27 (unknown):
```unknown
simulating ETV6
```

Example 28 (unknown):
```unknown
2025-05-12 12:15:33,121 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-12 12:15:33,139 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
 44%|████▍     | 4/9 [23:43<29:48, 357.62s/it]
```

Example 29 (unknown):
```unknown
simulating BHLHE41
```

Example 30 (unknown):
```unknown
2025-05-12 12:21:18,159 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-12 12:21:18,185 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
 56%|█████▌    | 5/9 [29:29<23:32, 353.20s/it]
```

Example 31 (unknown):
```unknown
simulating GTF2IRD1
```

Example 32 (unknown):
```unknown
2025-05-12 12:27:09,019 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-12 12:27:09,033 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
 67%|██████▋   | 6/9 [35:19<17:36, 352.29s/it]
```

Example 33 (unknown):
```unknown
simulating GATA5
```

Example 34 (unknown):
```unknown
2025-05-12 12:31:14,606 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-12 12:31:14,617 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
 78%|███████▊  | 7/9 [39:25<10:34, 317.39s/it]
```

Example 35 (unknown):
```unknown
simulating PITX2
```

Example 36 (unknown):
```unknown
2025-05-12 12:35:16,104 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-12 12:35:16,115 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
 89%|████████▉ | 8/9 [43:26<04:53, 293.21s/it]
```

Example 37 (unknown):
```unknown
simulating SIX5
```

Example 38 (unknown):
```unknown
2025-05-12 12:39:20,904 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
2025-05-12 12:39:20,928 - INFO - Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
100%|██████████| 9/9 [47:31<00:00, 316.84s/it]
```

Example 39 (unknown):
```unknown
# save
utils.save_pkl(dev_dict, path_to_res_dict)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/multiVI_scFate/no2_run_MultiVI_vCM_Left.html

**Contents:**
- Import modules¶
- Read in data¶
- Setup and training MultiVI (with continuous covariates)¶
  - All region, add 'region' key as a covariate¶

16451 rows × 33 columns

544641 rows × 21 columns

**Examples:**

Example 1 (python):
```python
# this is for the jupyter notebook at JupyterLab
# to make sys.path between the terminal and Jupyter same
import sys

print(sys.path) # especially the path to site-packages are different... so thats why couldn't find correct package...

# sys.path from terminal at scvi0190_env
# obtained by excuting `python -c "import sys; print('\n'.join(sys.path))"`
sys.path = [
    "/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python39.zip",
    "/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9",
    "/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/lib-dynload",
    "/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/site-packages"
]

print(sys.path)
```

Example 2 (unknown):
```unknown
['/nfs/team205/kk18/Analysis/multiVI/notebooks', '/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python39.zip', '/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9', '/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/lib-dynload', '', '/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/site-packages']
['/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python39.zip', '/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9', '/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/lib-dynload', '/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/site-packages']
```

Example 3 (python):
```python
import scvi
import anndata
import numpy as np
import pandas as pd
import scanpy as sc
import matplotlib.pyplot as plt

scvi.settings.seed = 420
```

Example 4 (python):
```python
Global seed set to 0
/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/site-packages/flax/core/frozen_dict.py:169: FutureWarning: jax.tree_util.register_keypaths is deprecated, and will be removed in a future release. Please use `register_pytree_with_keys()` instead.
  jax.tree_util.register_keypaths(
/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/site-packages/pytorch_lightning/utilities/warnings.py:53: LightningDeprecationWarning: pytorch_lightning.utilities.warnings.rank_zero_deprecation has been deprecated in v1.6 and will be removed in v1.8. Use the equivalent function from the pytorch_lightning.utilities.rank_zero module instead.
  new_rank_zero_deprecation(
/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/site-packages/pytorch_lightning/utilities/warnings.py:58: LightningDeprecationWarning: The `pytorch_lightning.loggers.base.rank_zero_experiment` is deprecated in v1.7 and will be removed in v1.9. Please use `pytorch_lightning.loggers.logger.rank_zero_experiment` instead.
  return new_rank_zero_deprecation(*args, **kwargs)
Global seed set to 420
```

Example 5 (python):
```python
import session_info
session_info.show()
```

Example 6 (python):
```python
-----
anndata             0.8.0
matplotlib          3.7.1
numpy               1.23.5
pandas              1.5.3
scanpy              1.9.3
scvi                0.19.0
session_info        1.0.0
-----
```

Example 7 (python):
```python
PIL                 9.4.0
absl                NA
asttokens           NA
attr                22.2.0
backcall            0.2.0
certifi             2022.12.07
cffi                1.15.1
charset_normalizer  3.1.0
chex                0.1.6
comm                0.1.2
contextlib2         NA
cycler              0.10.0
cython_runtime      NA
dateutil            2.8.2
debugpy             1.6.6
decorator           5.1.1
defusedxml          0.7.1
deprecate           0.3.2
docrep              0.3.2
etils               1.1.0
executing           1.2.0
flax                0.6.7
fsspec              2023.3.0
google              NA
h5py                3.8.0
idna                3.4
igraph              0.10.4
importlib_metadata  NA
importlib_resources NA
ipykernel           6.21.3
ipython_genutils    0.2.0
ipywidgets          8.0.4
jax                 0.4.10
jaxlib              0.4.10
jedi                0.18.2
joblib              1.2.0
kiwisolver          1.4.4
leidenalg           0.9.1
llvmlite            0.39.1
ml_collections      NA
ml_dtypes           0.2.0
mpl_toolkits        NA
msgpack             1.0.5
mudata              0.2.1
multipledispatch    0.6.0
natsort             8.3.1
numba               0.56.4
numpyro             0.11.0
nvidia              NA
opt_einsum          v3.3.0
optax               0.1.4
packaging           23.0
parso               0.8.3
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
platformdirs        3.1.1
prompt_toolkit      3.0.38
psutil              5.9.4
ptyprocess          0.7.0
pure_eval           0.2.2
pydev_ipython       NA
pydevconsole        NA
pydevd              2.9.5
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.14.0
pyparsing           3.0.9
pyro                1.8.4
pytorch_lightning   1.7.7
pytz                2022.7.1
requests            2.28.2
rich                NA
scipy               1.10.1
setuptools          65.6.3
six                 1.16.0
sklearn             1.2.2
stack_data          0.6.2
tensorboard         2.12.0
texttable           1.6.7
threadpoolctl       3.1.0
toolz               0.12.0
torch               1.13.1+cu117
torchmetrics        0.11.4
torchvision         0.14.1+cu117
tornado             6.2
tqdm                4.65.0
traitlets           5.9.0
tree                0.1.8
typing_extensions   NA
urllib3             1.26.15
wcwidth             0.2.6
yaml                6.0
zipp                NA
zmq                 25.0.1
zoneinfo            NA
```

Example 8 (unknown):
```unknown
-----
IPython             8.11.0
jupyter_client      8.0.3
jupyter_core        5.2.0
notebook            6.5.3
-----
Python 3.9.16 (main, Mar  8 2023, 14:00:05) [GCC 11.2.0]
Linux-4.15.0-213-generic-x86_64-with-glibc2.27
-----
Session information updated at 2023-08-15 23:34
```

Example 9 (python):
```python
# paired
# only multiome
adata_mvi = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/vCM/MultiVI-pre_paired_foetal_vCMs_raw.h5ad')
adata_mvi
```

Example 10 (unknown):
```unknown
AnnData object with n_obs × n_vars = 35705 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
```

Example 11 (python):
```python
adata_mvi.obs['fine_grain'].value_counts()
```

Example 12 (unknown):
```unknown
VentricularCardiomyocytesRightCompact         12008
VentricularCardiomyocytesLeftCompact          11397
VentricularCardiomyocytesLeftTrabeculated      5054
VentricularCardiomyocytesCycling               4977
VentricularCardiomyocytesRightTrabeculated     2269
Name: fine_grain, dtype: int64
```

Example 13 (python):
```python
# remove cycling population
adata_mvi = adata_mvi[adata_mvi.obs['fine_grain'].isin([
    'VentricularCardiomyocytesLeftCompact','VentricularCardiomyocytesLeftTrabeculated'
])]
adata_mvi.obs['fine_grain'].value_counts()
```

Example 14 (unknown):
```unknown
VentricularCardiomyocytesLeftCompact         11397
VentricularCardiomyocytesLeftTrabeculated     5054
Name: fine_grain, dtype: int64
```

Example 15 (python):
```python
adata_mvi.obs
```

Example 16 (python):
```python
adata_mvi.var['modality'].value_counts()
```

Example 17 (unknown):
```unknown
Peaks              508040
Gene Expression     36601
Name: modality, dtype: int64
```

Example 18 (python):
```python
adata_mvi.obs['modality'].value_counts()
```

Example 19 (unknown):
```unknown
Multiome    16451
Name: modality, dtype: int64
```

Example 20 (python):
```python
# MultiVI requires the features to be ordered, 
# such that genes appear before genomic regions. In this case this is already the case, but it’s always good to verify:
adata_mvi = adata_mvi[:, adata_mvi.var["modality"].argsort()].copy()
adata_mvi.var
```

Example 21 (python):
```python
adata_mvi_sub = adata_mvi.copy()
```

Example 22 (python):
```python
# We also filter features to remove those that appear in fewer than 1% of the cells
print(adata_mvi_sub.shape)
sc.pp.filter_genes(adata_mvi_sub, min_cells=int(adata_mvi_sub.shape[0] * 0.01))
print(adata_mvi_sub.shape)
```

Example 23 (unknown):
```unknown
(16451, 544641)
(16451, 174604)
```

Example 24 (python):
```python
adata_mvi_sub.var['modality'].value_counts()
```

Example 25 (unknown):
```unknown
Peaks              161398
Gene Expression     13206
Name: modality, dtype: int64
```

Example 26 (python):
```python
adata_mvi_sub.obs['modality'].value_counts()
```

Example 27 (unknown):
```unknown
Multiome    16451
Name: modality, dtype: int64
```

Example 28 (python):
```python
pd.crosstab(adata_mvi_sub.obs['batch_key'],adata_mvi_sub.obs['region'])
```

Example 29 (python):
```python
scvi.model.MULTIVI.setup_anndata(adata_mvi_sub, batch_key='modality',
                                categorical_covariate_keys=['batch_key','region'], ### adding 'region' ###
                                continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo',
                                                          'S_score','G2M_score'])
```

Example 30 (unknown):
```unknown
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
```

Example 31 (python):
```python
# When creating the object, we need to specify how many of the features are genes, and how many are genomic regions. 
# This is so MultiVI can determine the exact architecture for each modality.
mvi = scvi.model.MULTIVI(
    adata_mvi_sub,
    n_genes=(adata_mvi_sub.var['modality']=='Gene Expression').sum(),
    n_regions=(adata_mvi_sub.var['modality']=='Peaks').sum(),
)
mvi.view_anndata_setup()
```

Example 32 (unknown):
```unknown
/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
  warnings.warn("Initializing zero-element tensors is a no-op")
```

Example 33 (unknown):
```unknown
Anndata setup with scvi-tools version 0.19.0.
```

Example 34 (unknown):
```unknown
Setup via `MULTIVI.setup_anndata` with arguments:
```

Example 35 (unknown):
```unknown
{
│   'layer': None,
│   'batch_key': 'modality',
│   'size_factor_key': None,
│   'categorical_covariate_keys': ['batch_key', 'region'],
│   'continuous_covariate_keys': ['total_counts', 'pct_counts_mt', 'pct_counts_ribo', 'S_score', 'G2M_score'],
│   'protein_expression_obsm_key': None,
│   'protein_names_uns_key': None
}
```

Example 36 (unknown):
```unknown
Summary Statistics          
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃     Summary Stat Key     ┃ Value  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│         n_batch          │   1    │
│         n_cells          │ 16451  │
│ n_extra_categorical_covs │   2    │
│ n_extra_continuous_covs  │   5    │
│         n_labels         │   1    │
│          n_vars          │ 174604 │
└──────────────────────────┴────────┘
```

Example 37 (python):
```python
Data Registry                             
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃      Registry Key      ┃            scvi-tools Location             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│           X            │                  adata.X                   │
│         batch          │          adata.obs['_scvi_batch']          │
│ extra_categorical_covs │ adata.obsm['_scvi_extra_categorical_covs'] │
│ extra_continuous_covs  │ adata.obsm['_scvi_extra_continuous_covs']  │
│         ind_x          │           adata.obs['_indices']            │
│         labels         │         adata.obs['_scvi_labels']          │
└────────────────────────┴────────────────────────────────────────────┘
```

Example 38 (python):
```python
batch State Registry                    
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃    Source Location    ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['modality'] │  Multiome  │          0          │
└───────────────────────┴────────────┴─────────────────────┘
```

Example 39 (python):
```python
labels State Registry                      
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃      Source Location      ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['_scvi_labels'] │     0      │          0          │
└───────────────────────────┴────────────┴─────────────────────┘
```

Example 40 (python):
```python
batch State Registry                    
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃    Source Location    ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['modality'] │  Multiome  │          0          │
└───────────────────────┴────────────┴─────────────────────┘
```

Example 41 (python):
```python
extra_categorical_covs State Registry                       
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃    Source Location     ┃           Categories            ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['batch_key'] │     C82_Nuclei_Multiome-v1      │          0          │
│                        │     C85_Nuclei_Multiome-v1      │          1          │
│                        │     C87_Nuclei_Multiome-v1      │          2          │
│                        │     C92_Nuclei_Multiome-v1      │          3          │
│                        │     C98_Nuclei_Multiome-v1      │          4          │
│                        │     C104_Nuclei_Multiome-v1     │          5          │
│                        │    Hst33_Nuclei_Multiome-v1     │          6          │
│                        │    Hst36_Nuclei_Multiome-v1     │          7          │
│                        │    Hst39_Nuclei_Multiome-v1     │          8          │
│                        │    Hst40_Nuclei_Multiome-v1     │          9          │
│                        │    Hst41_Nuclei_Multiome-v1     │         10          │
│                        │                                 │                     │
│  adata.obs['region']   │              apex               │          0          │
│                        │              atria              │          1          │
│                        │          atrial septum          │          2          │
│                        │          whole sample           │          3          │
│                        │       heart without node        │          4          │
│                        │         left ventricle          │          5          │
│                        │              node               │          6          │
│                        │          outflow tract          │          7          │
│                        │         right ventricle         │          8          │
│                        │ right ventricle and papillaries │          9          │
│                        │                                 │                     │
└────────────────────────┴─────────────────────────────────┴─────────────────────┘
```

Example 42 (python):
```python
extra_continuous_covs State   
            Registry            
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃       Source Location        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│  adata.obs['total_counts']   │
│  adata.obs['pct_counts_mt']  │
│ adata.obs['pct_counts_ribo'] │
│     adata.obs['S_score']     │
│    adata.obs['G2M_score']    │
└──────────────────────────────┘
```

Example 43 (unknown):
```unknown
mvi.train()
```

Example 44 (unknown):
```unknown
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/nfs/team205/kk18/miniconda3/envs/scvi0190_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:267: LightningDeprecationWarning: The `Callback.on_epoch_end` hook was deprecated in v1.6 and will be removed in v1.8. Please use `Callback.on_<train/validation/test>_epoch_end` instead.
  rank_zero_deprecation(
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 45 (unknown):
```unknown
Epoch 269/500:  54%|██████████████████████                   | 269/500 [1:57:00<1:40:28, 26.10s/it, loss=1.42e+04, v_num=1]
Monitored metric reconstruction_loss_validation did not improve in the last 50 records. Best score: 30056.787. Signaling Trainer to stop.
```

Example 46 (unknown):
```unknown
mvi.save("/nfs/team205/heart/anndata_objects/Foetal/vCM/MultiVI-model_foetal_vCM_onlyMultiome_allregion_onlyLeft_feature-select-001")
```

Example 47 (python):
```python
adata_mvi_sub.obsm["MultiVI_latent"] = mvi.get_latent_representation()
```

Example 48 (python):
```python
adata_mvi_sub.write('/nfs/team205/heart/anndata_objects/Foetal/vCM/MultiVI-post_foetal_vCM_onlyMultiome_allregion_onlyLeft_feature-select-001.h5ad')
```

Example 49 (python):
```python
# Convergency plot
mvi.history['elbo_train']
x = np.linspace(1, (len(mvi.history['elbo_train'])), (len(mvi.history['elbo_train'])))
plt.plot(x, mvi.history['elbo_train'], label="train")
plt.xlabel('epoch')
plt.ylabel('elbo_train')
plt.show()
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/multiVI_scFate/no1_MultiVI_prep.html

**Contents:**
- Variables¶
- Read in¶
- Prepare paired anndata¶
- Prepare RNA data¶

35705 rows × 18 columns

36601 rows × 3 columns

**Examples:**

Example 1 (python):
```python
import scanpy as sc
import pandas as pd
import numpy as np
import anndata
import scipy

from scipy.io import mmwrite, mmread
from scipy.sparse import csr_matrix

import sctk
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (unknown):
```unknown
-----
anndata             0.8.0
numpy               1.21.5
pandas              1.3.5
scanpy              1.8.2
scipy               1.7.3
sctk                NA
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                 9.1.1
asciitree           NA
backcall            0.2.0
beta_ufunc          NA
binom_ufunc         NA
cached_property     1.5.1
cairo               1.20.0
cffi                1.14.4
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.2
dask                2022.01.0
dateutil            2.8.1
decorator           4.4.2
defusedxml          0.6.0
fasteners           0.17.3
fsspec              2022.01.0
google              NA
h5py                3.1.0
igraph              0.10.2
importlib_metadata  1.7.0
ipykernel           5.4.2
ipython_genutils    0.2.0
ipywidgets          7.6.3
jedi                0.17.2
jinja2              3.1.2
joblib              1.0.0
kiwisolver          1.3.1
leidenalg           0.9.0
llvmlite            0.35.0
louvain             0.7.0
markupsafe          2.1.1
matplotlib          3.5.2
mpl_toolkits        NA
msgpack             1.0.3
natsort             7.1.0
nbinom_ufunc        NA
numba               0.52.0
numcodecs           0.9.1
numexpr             2.7.2
numpy_groupies      0.9.13
packaging           21.3
parso               0.7.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
plotly              5.10.0
prompt_toolkit      3.0.10
psutil              5.9.0
ptyprocess          0.7.0
pyarrow             9.0.0
pygments            2.13.0
pyparsing           2.4.7
pytz                2020.5
scvelo              0.2.4
seaborn             0.11.1
setuptools_scm      NA
sinfo               0.3.1
six                 1.14.0
sklearn             1.0.2
sphinxcontrib       NA
statsmodels         0.12.1
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
threadpoolctl       2.1.0
tlz                 0.11.2
toolz               0.11.2
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
wcwidth             0.2.5
yaml                5.4.1
zarr                2.11.3
zipp                NA
zmq                 21.0.1
zope                NA
```

Example 5 (unknown):
```unknown
-----
IPython             7.19.0
jupyter_client      6.1.11
jupyter_core        4.7.0
notebook            6.2.0
-----
Python 3.7.9 | packaged by conda-forge | (default, Dec  9 2020, 21:08:20) [GCC 9.3.0]
Linux-4.15.0-112-generic-x86_64-with-debian-bullseye-sid
-----
Session information updated at 2023-08-10 21:16
```

Example 6 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 7 (unknown):
```unknown
cells_to_analyse = ['VentricularCardiomyocytesLeftCompact',
                     'VentricularCardiomyocytesLeftTrabeculated',
                    'VentricularCardiomyocytesRightCompact',
                     'VentricularCardiomyocytesRightTrabeculated',
                    'VentricularCardiomyocytesCycling']

celltype_key = 'fine_grain'

# region_to_analyse = ['whole sample']
```

Example 8 (unknown):
```unknown
!ls /nfs/team205/heart/anndata_objects/Foetal
```

Example 9 (python):
```python
14Feb_fromSemih		      QC
archive			      RNA_foetal_post-scVI_raw.h5ad
EC			      RNA_foetal_raw.h5ad
Feb28ObjectRaw.h5ad	      scArches
Feb28ObjectRaw_Multiome.h5ad  scVI
Feb28ObjectRaw_Multiome.rds   TipAndStalkAndGVSMCsPartitionedForCell2Loc.h5ad
Feb28ObjectRaw.obs.csv	      trisomy21
Feb28ObjectRaw_original.h5ad  visium-FFPE_foetal_raw.h5ad
Feb28ObjectRaw_re-scVI.h5ad   visium_foetal_healthy-all_raw.h5ad
FetalAtlas.h5ad		      visium_foetal_healthy-all_spot-annotation.csv
He_Cell_foetal-lung	      visium_foetal_raw.h5ad
Hill2022.h5ad		      visium-OCT_foetal_acrossage_raw.h5ad
Hill2022_QCapplied.h5ad       visium-OCT_foetal_DuctusA_raw.h5ad
Immune			      visium-OCT_foetal_raw.h5ad
metadata		      visium-OCT-LV_foetal_raw.h5ad
multiome_ATAC		      visium-OCT-T21_foetal_raw.h5ad
multiVI
```

Example 10 (python):
```python
# rna
adata_rna = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw.h5ad')
print(adata_rna.obs[celltype_key].value_counts())

# subset cells
adata_rna = adata_rna[adata_rna.obs[celltype_key].isin(cells_to_analyse)]

# subset regions
# adata_rna = adata_rna[adata_rna.obs['region'].isin(region_to_analyse)]

# add modality to .var
adata_rna.var['modality']='Gene Expression'

adata_rna
```

Example 11 (unknown):
```unknown
GreatVesselAdventitialFibroblasts        61593
GreatVesselSmoothMuscleCells             22992
VentricularCardiomyocytesRightCompact    16215
VentricularCardiomyocytesLeftCompact     15591
MyocardialInterstitialFibroblasts1       13880
                                         ...  
MacrophagesTIMD4pos                        470
MonocytesMPOpos                            414
Megakaryocytes                             360
ChromaffinCells                            228
DendriticCellsMature                       184
Name: fine_grain, Length: 63, dtype: int64
```

Example 12 (unknown):
```unknown
AnnData object with n_obs × n_vars = 50099 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id', 'modality'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 13 (python):
```python
pd.crosstab(adata_rna.obs[celltype_key],adata_rna.obs['region'])
```

Example 14 (python):
```python
pd.crosstab(adata_rna.obs[celltype_key],adata_rna.obs['kit_10x'])
```

Example 15 (python):
```python
pd.crosstab(adata_rna.obs[celltype_key],adata_rna.obs['cell_or_nuclei'])
```

Example 16 (python):
```python
pd.crosstab(adata_rna.obs['age'],adata_rna.obs['cell_or_nuclei'])
```

Example 17 (python):
```python
pd.crosstab(adata_rna.obs[celltype_key],adata_rna.obs['cycling'])
```

Example 18 (python):
```python
adata_rna.obs['kit_10x'].value_counts()
```

Example 19 (unknown):
```unknown
Multiome-v1    41279
5prime-v2       4788
3prime-v3       4032
Name: kit_10x, dtype: int64
```

Example 20 (python):
```python
# atac
# ie only Multiome data
adata_atac = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/multiome_ATAC/ArchR/project_output/PeakMatrix/Foetal_Peaks.h5ad')

# subset cells
adata_atac = adata_atac[adata_atac.obs[celltype_key].isin(cells_to_analyse)]
print(adata_atac.obs[celltype_key].value_counts())

# subset regions
# adata_atac = adata_atac[adata_atac.obs['region'].isin(region_to_analyse)]

# add modality to .var
adata_atac.var['modality']='Peaks'

adata_atac
```

Example 21 (unknown):
```unknown
VentricularCardiomyocytesRightCompact         12008
VentricularCardiomyocytesLeftCompact          11397
VentricularCardiomyocytesLeftTrabeculated      5054
VentricularCardiomyocytesCycling               4977
VentricularCardiomyocytesRightTrabeculated     2269
Name: fine_grain, dtype: int64
```

Example 22 (unknown):
```unknown
AnnData object with n_obs × n_vars = 35705 × 508040
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels'
    var: 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N', 'modality'
```

Example 23 (python):
```python
pd.crosstab(adata_atac.obs[celltype_key],adata_atac.obs['region'])
```

Example 24 (python):
```python
adata_atac.obs
```

Example 25 (python):
```python
# shared barcodes, ie Multiome data (not all, since MultiomeRNA has a bit more data than MultiomeATAC)
barcodes_inter=list(set(adata_rna.obs_names).intersection(adata_atac.obs_names))
len(barcodes_inter)

# take shared cells
multiome_rna=adata_rna[barcodes_inter,:]
multiome_atac=adata_atac[barcodes_inter,:]

# concatenate
adata_paired=anndata.AnnData(
    X=scipy.sparse.hstack([multiome_rna.X,multiome_atac.X]).tocsr(), # concatenate sparse matrix and convert to Compressed Sparse Row format
    obs=multiome_rna.obs[['sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility',
                           'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling',
                           'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain','coarse_grain','batch_key',
                          # ATAC QC

                          # RNA QC
                            'total_counts',
                     'total_counts_mt',
                     'total_counts_ribo',
                            'n_genes_by_counts',
                     'pct_counts_mt',
                     'pct_counts_ribo',
                            'log1p_n_genes_by_counts',
                     'log1p_total_counts',
                     'log1p_total_counts_mt',
                     'log1p_total_counts_ribo',
                     'scrublet_score',
                        'doublet_pval',
                        'doublet_bh_pval',]],
    var=pd.concat([multiome_rna.var[['modality','gene_name_scRNA-0','gene_id']], \
                   multiome_atac.var
                  ],axis=0)
)
adata_paired
```

Example 26 (unknown):
```unknown
AnnData object with n_obs × n_vars = 35705 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
```

Example 27 (python):
```python
# modify modality columns in .obs
adata_paired.obs['modality'] = 'Multiome'
adata_paired.obs['modality'].value_counts()
```

Example 28 (unknown):
```unknown
Multiome    35705
Name: modality, dtype: int64
```

Example 29 (python):
```python
# save
adata_paired.write('/nfs/team205/heart/anndata_objects/Foetal/vCM/MultiVI-pre_paired_foetal_vCMs_raw.h5ad')
```

Example 30 (python):
```python
adata_paired.obs.head()
```

Example 31 (python):
```python
adata_paired.obs['batch_key'].value_counts()
```

Example 32 (unknown):
```unknown
Hst33_Nuclei_Multiome-v1    17373
Hst41_Nuclei_Multiome-v1     9591
Hst36_Nuclei_Multiome-v1     2152
Hst39_Nuclei_Multiome-v1     1932
C87_Nuclei_Multiome-v1       1440
Hst40_Nuclei_Multiome-v1     1429
C104_Nuclei_Multiome-v1       720
C98_Nuclei_Multiome-v1        421
C92_Nuclei_Multiome-v1        322
C85_Nuclei_Multiome-v1        280
C82_Nuclei_Multiome-v1         45
Name: batch_key, dtype: int64
```

Example 33 (python):
```python
# select single-cell data (3p and 5p)
adata_scrna = adata_rna[adata_rna.obs['kit_10x'].isin(['3prime-v3','5prime-v2'])]
adata_scrna.obs['kit_10x'].value_counts()
```

Example 34 (unknown):
```unknown
5prime-v2    4788
3prime-v3    4032
Name: kit_10x, dtype: int64
```

Example 35 (python):
```python
adata_scrna.X.data[:10]
```

Example 36 (unknown):
```unknown
array([6., 1., 3., 1., 1., 1., 8., 3., 1., 1.], dtype=float32)
```

Example 37 (python):
```python
# recreate anndata
adata_scrna=anndata.AnnData(
    X=adata_scrna.X, # concatenate sparse matrix and convert to Compressed Sparse Row format
    obs=adata_scrna.obs[['sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility',
                           'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling',
                           'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain','coarse_grain','batch_key',
                          # ATAC QC

                          # RNA QC
                            'total_counts',
                     'total_counts_mt',
                     'total_counts_ribo',
                            'n_genes_by_counts',
                     'pct_counts_mt',
                     'pct_counts_ribo',
                            'log1p_n_genes_by_counts',
                     'log1p_total_counts',
                     'log1p_total_counts_mt',
                     'log1p_total_counts_ribo',
                     'scrublet_score',
                        'doublet_pval',
                        'doublet_bh_pval',]],
    var=adata_scrna.var[['modality','gene_name_scRNA-0','gene_id']]
)
adata_scrna
```

Example 38 (unknown):
```unknown
AnnData object with n_obs × n_vars = 8820 × 36601
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
```

Example 39 (python):
```python
# modify modality columns in .obs
adata_scrna.obs['modality'] = 'scRNA'
adata_scrna.obs['modality'].value_counts()
```

Example 40 (unknown):
```unknown
scRNA    8820
Name: modality, dtype: int64
```

Example 41 (python):
```python
adata_scrna.var
```

Example 42 (python):
```python
# save
adata_scrna.write('/nfs/team205/heart/anndata_objects/Foetal/vCM/MultiVI-pre_scRNA_foetal_vCMs_raw.h5ad')
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/CellOracle/CellOracle_no3_compact-common-up_summary.html

**Contents:**
- Import libraries¶
- Parameters¶
- Read in celloracle in-silico perturbation output¶
- Plot inner product scores¶
- Correlation of centrality and total inner product (effect of the perturbation)¶
- Read in CHD genes¶
- Project PS score on TF-TF network¶
- Plot representative TFs¶

ref: https://morris-lab.github.io/CellOracle.documentation/notebooks/04_Network_analysis/Network_analysis_with_Paul_etal_2015_data.html

**Examples:**

Example 1 (python):
```python
import os
import sys

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scanpy as sc
import seaborn as sns

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42
```

Example 2 (python):
```python
import celloracle as co
co.__version__
```

Example 3 (python):
```python
/nfs/team205/kk18/miniconda3/envs/celloracle_env/lib/python3.8/site-packages/loompy/bus_file.py:68: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def twobit_to_dna(twobit: int, size: int) -> str:
/nfs/team205/kk18/miniconda3/envs/celloracle_env/lib/python3.8/site-packages/loompy/bus_file.py:85: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def dna_to_twobit(dna: str) -> int:
/nfs/team205/kk18/miniconda3/envs/celloracle_env/lib/python3.8/site-packages/loompy/bus_file.py:102: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def twobit_1hamming(twobit: int, size: int) -> List[int]:
```

Example 4 (python):
```python
import session_info
session_info.show()
```

Example 5 (python):
```python
-----
celloracle          0.14.0
matplotlib          3.6.3
numpy               1.24.3
pandas              1.5.3
scanpy              1.9.5
seaborn             0.13.0
session_info        1.0.0
-----
```

Example 6 (python):
```python
Bio                         1.81
PIL                         10.0.1
anndata                     0.9.2
anyio                       NA
appdirs                     1.4.4
arrow                       1.3.0
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.13.0
backcall                    0.2.0
beta_ufunc                  NA
binom_ufunc                 NA
biothings_client            0.3.0
bottleneck                  1.3.5
brotli                      NA
certifi                     2023.07.22
cffi                        1.15.1
charset_normalizer          2.0.4
colorama                    0.4.6
comm                        0.1.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.8.0
decorator                   5.1.1
defusedxml                  0.7.1
diskcache                   5.6.3
exceptiongroup              1.1.3
executing                   2.0.0
fastjsonschema              NA
filelock                    3.12.4
fqdn                        NA
genomepy                    0.16.1
gimmemotifs                 0.17.0
goatools                    1.3.9
h5py                        3.10.0
hypergeom_ufunc             NA
idna                        3.4
igraph                      0.10.8
importlib_metadata          NA
importlib_resources         NA
invgauss_ufunc              NA
ipykernel                   6.25.2
ipywidgets                  8.1.1
isoduration                 NA
iteround                    NA
jedi                        0.19.1
jinja2                      3.1.2
joblib                      1.2.0
json5                       NA
jsonpointer                 2.4
jsonschema                  4.19.1
jsonschema_specifications   NA
jupyter_events              0.8.0
jupyter_server              2.8.0
jupyterlab_server           2.25.0
kiwisolver                  1.4.4
llvmlite                    0.41.0
logomaker                   NA
loguru                      0.7.2
loompy                      3.0.7
louvain                     0.8.0
markupsafe                  2.1.3
mkl                         2.4.0
mpl_toolkits                NA
mygene                      3.2.2
mysql                       NA
natsort                     8.4.0
nbformat                    5.9.2
nbinom_ufunc                NA
ncf_ufunc                   NA
nct_ufunc                   NA
ncx2_ufunc                  NA
networkx                    3.1
norns                       NA
numba                       0.58.0
numexpr                     2.8.4
numpy_groupies              0.9.22
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
platformdirs                3.10.0
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.6
ptyprocess                  0.7.0
pure_eval                   0.2.2
pyarrow                     13.0.0
pybedtools                  0.9.1
pycparser                   2.21
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pyfaidx                     0.7.2.2
pygments                    2.16.1
pyparsing                   3.0.9
pysam                       0.20.0
pythonjsonlogger            NA
pytz                        2023.3.post1
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
scipy                       1.10.1
send2trash                  NA
setuptools                  68.2.2
six                         1.16.0
skewnorm_ufunc              NA
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
stack_data                  0.6.3
statsmodels                 0.14.0
texttable                   1.7.0
threadpoolctl               2.2.0
tornado                     6.3.3
tqdm                        4.66.1
traitlets                   5.11.2
typing_extensions           NA
uri_template                NA
urllib3                     1.26.16
velocyto                    0.17.17
wcwidth                     0.2.8
webcolors                   1.13
websocket                   1.6.4
xdg                         NA
xgboost                     2.0.0
xxhash                      NA
yaml                        6.0.1
zipp                        NA
zmq                         25.1.1
```

Example 7 (unknown):
```unknown
-----
IPython             8.12.3
jupyter_client      8.4.0
jupyter_core        5.4.0
jupyterlab          4.0.7
notebook            7.0.6
-----
Python 3.8.18 (default, Sep 11 2023, 13:40:15) [GCC 11.2.0]
Linux-5.4.0-169-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-03-25 12:47
```

Example 8 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/regulon_plot.py")
regulon_plot = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = regulon_plot
spec.loader.exec_module(regulon_plot)
```

Example 9 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 10 (python):
```python
%matplotlib inline
```

Example 11 (unknown):
```unknown
# Parameters
working_dir = "/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/CO_in-silico-perturbation_papermill"
path_to_oracle_object = "/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.celloracle.oracle"
path_to_oracle_links = "/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/vCM_onlyLeft_links.celloracle.links"
n_grid = 40
min_mass = 0.011
path_to_TFs = "/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/TF_selection_df_compact-common-up_2Feb2024.csv"
path_to_res_dict = "/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/vCM_onlyLeft_compact-common_silico_perturbation_2Feb2024.pkl"
```

Example 12 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables'
```

Example 13 (python):
```python
dev_dict = utils.read_pkl(path_to_res_dict)
print(len(dev_dict.keys()))
dev_dict.keys()
```

Example 14 (unknown):
```unknown
dict_keys(['ARID3A', 'ATF1', 'BACH2', 'BHLHE40', 'CUX1', 'ESRRA', 'ESRRB', 'FOXN3', 'FOXO1', 'GATA4', 'GATA6', 'GTF2IRD1', 'HAND2', 'HEY2', 'HIF1A', 'HMBOX1', 'KLF12', 'KLF6', 'LCORL', 'MAX', 'MAZ', 'MBNL2', 'MEF2A', 'MEF2C', 'MEF2D', 'MEIS2', 'MXI1', 'NCOA1', 'NFATC2', 'NFE2L1', 'NFIA', 'NFIB', 'NFIC', 'NPAS2', 'NR3C1', 'NR4A1', 'PBX1', 'PLAG1', 'PPARA', 'PRDM16', 'RORA', 'SMAD3', 'SREBF2', 'TBX20', 'TFCP2', 'TGIF2', 'THRA', 'USF2', 'YY1', 'ZBTB16', 'ZBTB20', 'ZNF148', 'ZNF280D', 'ZNF654', 'ZSCAN18', 'AHCTF1', 'CREB1', 'FOXK1', 'PHF21A', 'SOX5', 'UBP1', 'ZNF141', 'ELF2', 'POU2F1', 'ZNF586', 'IRX3'])
```

Example 15 (python):
```python
%matplotlib inline

vm = 0.01
scale_simulation = 0.1
scale_dev = 40

tf = 'PRDM16'
dev = dev_dict[tf]
dev.visualize_development_module_layout_0(s=5,
                                          scale_for_simulation=scale_simulation,
                                          s_grid=50,
                                          scale_for_pseudotime=scale_dev,
                                          vm=vm)
```

Example 16 (python):
```python
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
INFO:matplotlib.category:Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 17 (python):
```python
from scipy.interpolate import UnivariateSpline

nrows = 10
ncols = 7
fig,axs=plt.subplots(nrows=nrows,ncols=ncols,figsize=(25,25)) 
for i,tf in enumerate(dev_dict.keys()):
    dev = dev_dict[tf]
    # get inner product results
    inner_product_df = dev.inner_product_df.copy()
    # normalise pseudotime (min-max)
    arr = np.array(inner_product_df['pseudotime'])
    inner_product_df['pseudotime_norm'] = (arr-np.min(arr))/(np.max(arr)-np.min(arr))
    # sort
    inner_product_df.sort_values('pseudotime_norm',ascending=True,inplace=True)
    # smoothing inner product score
    spl_res = UnivariateSpline(inner_product_df['pseudotime_norm'], inner_product_df['score'], k=2)
    inner_product_df['score_smoothed'] = spl_res(inner_product_df['pseudotime_norm'])
    # plot
    axs[i//ncols,i%ncols].scatter(x=inner_product_df['pseudotime_norm'],y=inner_product_df['score'])
    axs[i//ncols,i%ncols].plot(inner_product_df['pseudotime_norm'],inner_product_df['score_smoothed'],color='red')
    axs[i//ncols,i%ncols].set_title(tf)
    axs[i//ncols,i%ncols].set_xlabel('pseudotime')
    axs[i//ncols,i%ncols].set_ylabel('inner product score')
# fig.suptitle('')
fig.tight_layout()
```

Example 18 (python):
```python
from scipy.stats import pearsonr
```

Example 19 (python):
```python
# raed in TF data
TF_df = pd.read_csv(path_to_TFs)
TF_df = TF_df.set_index('TF')

# get total inner product
total_inner_product = {}
for tf in dev_dict.keys():
    dev = dev_dict[tf]
    # get inner product results
    inner_product_df = dev.inner_product_df.copy()
    # take sum of the score
    TF_df.loc[tf,'total_PS_score'] = inner_product_df['score'].sum()

# save the dataframe
TF_df.to_csv(os.path.join(table_dir,'csvs/TF_metrics_vCM_onlyLeft.csv'))

TF_df
```

Example 20 (unknown):
```unknown
os.path.join(table_dir,'csvs/TF_metrics_vCM_onlyLeft.csv')
```

Example 21 (unknown):
```unknown
'/nfs/team205/heart/fetal_ms_figures/tables/csvs/TF_metrics_vCM_onlyLeft.csv'
```

Example 22 (python):
```python
r,p = pearsonr(TF_df['centaliry_eigenvector'],TF_df['total_PS_score'])
sns.regplot(data=TF_df, x='centaliry_eigenvector', y='total_PS_score')
plt.xlabel('centrality')
plt.ylabel('total PS score')
plt.title(f"Pearson's r: {round(r,3)}, p-value: {round(p,3)}")
plt.savefig(f"{figdir}/TF-network_compact-common-up_correlation-centrality-PS.pdf",bbox_inches="tight",dpi=300)
plt.show()
```

Example 23 (unknown):
```unknown
INFO:fontTools.subset:maxp pruned
INFO:fontTools.subset:cmap pruned
INFO:fontTools.subset:kern dropped
INFO:fontTools.subset:post pruned
INFO:fontTools.subset:FFTM dropped
INFO:fontTools.subset:GPOS pruned
INFO:fontTools.subset:GSUB pruned
INFO:fontTools.subset:name pruned
INFO:fontTools.subset:glyf pruned
INFO:fontTools.subset:Added gid0 to subset
INFO:fontTools.subset:Added first four glyphs to subset
INFO:fontTools.subset:Closing glyph list over 'GSUB': 32 glyphs before
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'P', 'S', 'a', 'c', 'colon', 'comma', 'e', 'five', 'four', 'hyphen', 'i', 'l', 'minus', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'period', 'quotesingle', 'r', 's', 'space', 't', 'three', 'two', 'u', 'v', 'y', 'zero']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 10, 15, 16, 17, 19, 20, 21, 22, 23, 24, 29, 51, 54, 68, 70, 72, 76, 79, 81, 82, 83, 85, 86, 87, 88, 89, 92, 3228]
INFO:fontTools.subset:Closed glyph list over 'GSUB': 32 glyphs after
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'P', 'S', 'a', 'c', 'colon', 'comma', 'e', 'five', 'four', 'hyphen', 'i', 'l', 'minus', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'period', 'quotesingle', 'r', 's', 'space', 't', 'three', 'two', 'u', 'v', 'y', 'zero']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 10, 15, 16, 17, 19, 20, 21, 22, 23, 24, 29, 51, 54, 68, 70, 72, 76, 79, 81, 82, 83, 85, 86, 87, 88, 89, 92, 3228]
INFO:fontTools.subset:Closing glyph list over 'MATH': 32 glyphs before
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'P', 'S', 'a', 'c', 'colon', 'comma', 'e', 'five', 'four', 'hyphen', 'i', 'l', 'minus', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'period', 'quotesingle', 'r', 's', 'space', 't', 'three', 'two', 'u', 'v', 'y', 'zero']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 10, 15, 16, 17, 19, 20, 21, 22, 23, 24, 29, 51, 54, 68, 70, 72, 76, 79, 81, 82, 83, 85, 86, 87, 88, 89, 92, 3228]
INFO:fontTools.subset:Closed glyph list over 'MATH': 32 glyphs after
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'P', 'S', 'a', 'c', 'colon', 'comma', 'e', 'five', 'four', 'hyphen', 'i', 'l', 'minus', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'period', 'quotesingle', 'r', 's', 'space', 't', 'three', 'two', 'u', 'v', 'y', 'zero']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 10, 15, 16, 17, 19, 20, 21, 22, 23, 24, 29, 51, 54, 68, 70, 72, 76, 79, 81, 82, 83, 85, 86, 87, 88, 89, 92, 3228]
INFO:fontTools.subset:Closing glyph list over 'glyf': 32 glyphs before
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'P', 'S', 'a', 'c', 'colon', 'comma', 'e', 'five', 'four', 'hyphen', 'i', 'l', 'minus', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'period', 'quotesingle', 'r', 's', 'space', 't', 'three', 'two', 'u', 'v', 'y', 'zero']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 10, 15, 16, 17, 19, 20, 21, 22, 23, 24, 29, 51, 54, 68, 70, 72, 76, 79, 81, 82, 83, 85, 86, 87, 88, 89, 92, 3228]
INFO:fontTools.subset:Closed glyph list over 'glyf': 32 glyphs after
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'P', 'S', 'a', 'c', 'colon', 'comma', 'e', 'five', 'four', 'hyphen', 'i', 'l', 'minus', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'period', 'quotesingle', 'r', 's', 'space', 't', 'three', 'two', 'u', 'v', 'y', 'zero']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 10, 15, 16, 17, 19, 20, 21, 22, 23, 24, 29, 51, 54, 68, 70, 72, 76, 79, 81, 82, 83, 85, 86, 87, 88, 89, 92, 3228]
INFO:fontTools.subset:Retaining 32 glyphs
INFO:fontTools.subset:head subsetting not needed
INFO:fontTools.subset:hhea subsetting not needed
INFO:fontTools.subset:maxp subsetting not needed
INFO:fontTools.subset:OS/2 subsetting not needed
INFO:fontTools.subset:hmtx subsetted
INFO:fontTools.subset:cmap subsetted
INFO:fontTools.subset:fpgm subsetting not needed
INFO:fontTools.subset:prep subsetting not needed
INFO:fontTools.subset:cvt  subsetting not needed
INFO:fontTools.subset:loca subsetting not needed
INFO:fontTools.subset:post subsetted
INFO:fontTools.subset:gasp subsetting not needed
INFO:fontTools.subset:GDEF subsetted
INFO:fontTools.subset:GPOS subsetted
INFO:fontTools.subset:GSUB subsetted
INFO:fontTools.subset:MATH subsetted
INFO:fontTools.subset:name subsetting not needed
INFO:fontTools.subset:glyf subsetted
INFO:fontTools.subset:head pruned
INFO:fontTools.subset:OS/2 Unicode ranges pruned: [0, 38]
INFO:fontTools.subset:glyf pruned
INFO:fontTools.subset:GDEF pruned
INFO:fontTools.subset:GPOS pruned
INFO:fontTools.subset:GSUB pruned
```

Example 24 (python):
```python
CHDs = pd.read_csv('./combined_tidied_disease_gene_with_phenotype_annotation_KKmod.csv',index_col=0)
print(CHDs.shape)
CHDs.head(n=3)
```

Example 25 (unknown):
```unknown
CHDs.loc['PRDM16']
```

Example 26 (unknown):
```unknown
TF                                                                                          1
source                                                                                ClinVar
phenotype_Heterotaxy                                                                        0
phenotype_ASD                                                                               0
phenotype_VSD                                                                               0
phenotype_AVSD                                                                              0
phenotype_Malformation of outflow tracts                                                    0
phenotype_Functional single ventricle                                                       0
phenotype_Cardiomyopathy                                                                    1
phenotype_DCM                                                                               0
phenotype_HCM                                                                               0
phenotype_LVNC                                                                              1
phenotype_Venous anomaly                                                                    0
phenotype_Pulmonary artery                                                                  0
phenotype_Aortic arch                                                                       0
phenotype_Aorta                                                                             0
phenotype_Coronary anomaly                                                                  0
phenotype_VA valves                                                                         0
phenotype_AV valves                                                                         0
phenotype                                   Left ventricular noncompaction 8|Left ventricu...
Name: PRDM16, dtype: object
```

Example 27 (python):
```python
# subset table based on phenotypes
phenotypes = ['phenotype_VSD','phenotype_AVSD','phenotype_LVNC','phenotype_Cardiomyopathy','phenotype_Functional single ventricle','phenotype_DCM','phenotype_HCM']
mask = CHDs[phenotypes].sum(axis=1)>0
CHDs_sel = CHDs[mask]
print(CHDs_sel.shape)
CHDs_sel.head(n=3)
```

Example 28 (unknown):
```unknown
CHDs_sel['TF'].value_counts()
```

Example 29 (unknown):
```unknown
0    2511
1      30
Name: TF, dtype: int64
```

Example 30 (python):
```python
# select TF list inplicated in the phenotypes
CHD_TFs = list(CHDs_sel[CHDs_sel['TF']==1].index)
print(len(CHD_TFs))
CHD_TFs[:5]
```

Example 31 (unknown):
```unknown
['FOXC2', 'FOXP1', 'GATA4', 'GATA5', 'GATA6']
```

Example 32 (unknown):
```unknown
# intersect with the TFs in the network
'PRDM16' in CHD_TFs
```

Example 33 (unknown):
```unknown
# read in TF-TF network
network_TF = utils.read_pkl('/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/TF-TF-network_compact-common-up_2Feb2024.csv')
len(network_TF.keys())
```

Example 34 (python):
```python
# prepare color map
# total PS score

from matplotlib.colors import TwoSlopeNorm

# vector of continuous values
values = np.array(TF_df['total_PS_score'])

# Find the maximum absolute value for normalization
max_abs_value = np.max(np.abs(values))
# Normalize the values manually to ensure zero is centered
# Map the range [-max_abs_value, max_abs_value] to [0, 1] with 0.5 at zero
normalized_values = 0.5 + (values / (2 * max_abs_value))

# Use a diverging colormap
cmap = plt.get_cmap('PiYG')
colors = cmap(normalized_values)

# make mapping dictionary
nodes_color_map = dict(zip(TF_df.index,colors))
```

Example 35 (unknown):
```unknown
# prepare parameters
size_scale = 7000

centrality_type = 'eigenvector'
centrality_dict = regulon_plot.get_centrality(network_TF,
                   directed=True,
                   centrality_type=centrality_type, # or 'degree','in_dgree','out_degree'
                   out_df=False # if False output dictionary
                  )

node_size_map = {}
for n in network_TF.keys():
    if n in centrality_dict.keys():
        node_size_map[n]=int(centrality_dict[n]*size_scale)
    else: # means no connection with other TFs
        node_size_map[n]=0
```

Example 36 (python):
```python
# select CHD TFs which are in the network constructed
all_TFs = []
for x in network_TF.values():
    all_TFs = all_TFs + x
all_TFs = all_TFs + list(network_TF.keys())

CHD_TFs_sel = list(set(CHD_TFs).intersection(all_TFs))
print(len(CHD_TFs_sel))
CHD_TFs_sel
```

Example 37 (unknown):
```unknown
['PBX1', 'HAND2', 'PRDM16', 'GATA6', 'SMAD3', 'TBX20', 'MEIS2', 'GATA4']
```

Example 38 (python):
```python
from matplotlib.colors import Normalize
from matplotlib.cm import ScalarMappable
from matplotlib.colorbar import ColorbarBase
from mpl_toolkits.axes_grid1 import make_axes_locatable

fig, ax = plt.subplots(figsize=(8,8))

regulon_plot.draw_networkx(
    network=network_TF,
    nodes_color_base='white',
    nodes_color_map=nodes_color_map,
    nodes_size_base=400,
    nodes_size_map=node_size_map,
    nodes_edge_highlight=CHD_TFs_sel,
    nodes_edge_color='blue',
    nodes_edge_thickness=3,
     spring_layout_k=1.5,
    spring_layout_scale=1,
    font_size=12,
    pos_seed=3,
    ax=ax
)

# Create an axes on the right side of ax. The width of cax will be 5%
# of ax and the padding between cax and ax will be fixed at 0.05 inch.
divider = make_axes_locatable(ax)
cbar_ax = divider.append_axes("right", size="2%", pad=0.01)
# cbar_ax = fig.add_axes([0.9, 0.1, 0.05, 0.3])

# Create a colorbar with correct scaling
# Setup the normalization and colormap for the colorbar
norm = Normalize(vmin=-max_abs_value, vmax=max_abs_value)
sm = ScalarMappable(norm=norm, cmap=cmap)
sm.set_array([])
cbar = plt.colorbar(sm, cax=cbar_ax, orientation='vertical')
cbar.set_label('Perturbation score (sum)')

fig.suptitle('TF network for compact CM maturation')
fig.tight_layout()
# fig.savefig(f"{figdir}/TF-network_compact-common-up.pdf",bbox_inches="tight",dpi=300)
```

Example 39 (unknown):
```unknown
<Figure size 1200x1200 with 0 Axes>
```

Example 40 (python):
```python
TFs_to_plot = ['HEY2','PRDM16','NFE2L1','SREBF2']

fig,axs=plt.subplots(nrows=1,ncols=len(TFs_to_plot),figsize=(14,4)) 
for i,tf in enumerate(TFs_to_plot):
    dev = dev_dict[tf]
    dev.plot_inner_product_on_grid(vm=vm, s=50, ax=axs[i])
    dev.plot_simulation_flow_on_grid(scale=scale_simulation, show_background=False, ax=axs[i])
    axs[i].set_title(tf)
fig.tight_layout(w_pad=-0.3)
fig.savefig(f"{figdir}/vCM-left_celloracle_PS.pdf",bbox_inches="tight",dpi=300)
```

Example 41 (unknown):
```unknown
INFO:fontTools.subset:maxp pruned
INFO:fontTools.subset:cmap pruned
INFO:fontTools.subset:kern dropped
INFO:fontTools.subset:post pruned
INFO:fontTools.subset:FFTM dropped
INFO:fontTools.subset:GPOS pruned
INFO:fontTools.subset:GSUB pruned
INFO:fontTools.subset:name pruned
INFO:fontTools.subset:glyf pruned
INFO:fontTools.subset:Added gid0 to subset
INFO:fontTools.subset:Added first four glyphs to subset
INFO:fontTools.subset:Closing glyph list over 'GSUB': 19 glyphs before
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'B', 'D', 'E', 'F', 'H', 'L', 'M', 'N', 'P', 'R', 'S', 'Y', 'nonmarkingreturn', 'one', 'six', 'space', 'two']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 20, 21, 25, 37, 39, 40, 41, 43, 47, 48, 49, 51, 53, 54, 60]
INFO:fontTools.subset:Closed glyph list over 'GSUB': 19 glyphs after
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'B', 'D', 'E', 'F', 'H', 'L', 'M', 'N', 'P', 'R', 'S', 'Y', 'nonmarkingreturn', 'one', 'six', 'space', 'two']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 20, 21, 25, 37, 39, 40, 41, 43, 47, 48, 49, 51, 53, 54, 60]
INFO:fontTools.subset:Closing glyph list over 'MATH': 19 glyphs before
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'B', 'D', 'E', 'F', 'H', 'L', 'M', 'N', 'P', 'R', 'S', 'Y', 'nonmarkingreturn', 'one', 'six', 'space', 'two']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 20, 21, 25, 37, 39, 40, 41, 43, 47, 48, 49, 51, 53, 54, 60]
INFO:fontTools.subset:Closed glyph list over 'MATH': 19 glyphs after
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'B', 'D', 'E', 'F', 'H', 'L', 'M', 'N', 'P', 'R', 'S', 'Y', 'nonmarkingreturn', 'one', 'six', 'space', 'two']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 20, 21, 25, 37, 39, 40, 41, 43, 47, 48, 49, 51, 53, 54, 60]
INFO:fontTools.subset:Closing glyph list over 'glyf': 19 glyphs before
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'B', 'D', 'E', 'F', 'H', 'L', 'M', 'N', 'P', 'R', 'S', 'Y', 'nonmarkingreturn', 'one', 'six', 'space', 'two']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 20, 21, 25, 37, 39, 40, 41, 43, 47, 48, 49, 51, 53, 54, 60]
INFO:fontTools.subset:Closed glyph list over 'glyf': 19 glyphs after
INFO:fontTools.subset:Glyph names: ['.notdef', '.null', 'B', 'D', 'E', 'F', 'H', 'L', 'M', 'N', 'P', 'R', 'S', 'Y', 'nonmarkingreturn', 'one', 'six', 'space', 'two']
INFO:fontTools.subset:Glyph IDs:   [0, 1, 2, 3, 20, 21, 25, 37, 39, 40, 41, 43, 47, 48, 49, 51, 53, 54, 60]
INFO:fontTools.subset:Retaining 19 glyphs
INFO:fontTools.subset:head subsetting not needed
INFO:fontTools.subset:hhea subsetting not needed
INFO:fontTools.subset:maxp subsetting not needed
INFO:fontTools.subset:OS/2 subsetting not needed
INFO:fontTools.subset:hmtx subsetted
INFO:fontTools.subset:cmap subsetted
INFO:fontTools.subset:fpgm subsetting not needed
INFO:fontTools.subset:prep subsetting not needed
INFO:fontTools.subset:cvt  subsetting not needed
INFO:fontTools.subset:loca subsetting not needed
INFO:fontTools.subset:post subsetted
INFO:fontTools.subset:gasp subsetting not needed
INFO:fontTools.subset:GDEF subsetted
INFO:fontTools.subset:GPOS subsetted
INFO:fontTools.subset:GSUB subsetted
INFO:fontTools.subset:MATH subsetted
INFO:fontTools.subset:name subsetting not needed
INFO:fontTools.subset:glyf subsetted
INFO:fontTools.subset:head pruned
INFO:fontTools.subset:OS/2 Unicode ranges pruned: [0]
INFO:fontTools.subset:glyf pruned
INFO:fontTools.subset:GDEF pruned
INFO:fontTools.subset:GPOS pruned
INFO:fontTools.subset:GSUB pruned
```

Example 42 (unknown):
```unknown
?fig.tight_layout
```

Example 43 (python):
```python
Signature: fig.tight_layout(*, pad=1.08, h_pad=None, w_pad=None, rect=None)
Docstring:
Adjust the padding between and around subplots.

To exclude an artist on the Axes from the bounding box calculation
that determines the subplot parameters (i.e. legend, or annotation),
set ``a.set_in_layout(False)`` for that artist.

Parameters
----------
pad : float, default: 1.08
    Padding between the figure edge and the edges of subplots,
    as a fraction of the font size.
h_pad, w_pad : float, default: *pad*
    Padding (height/width) between edges of adjacent subplots,
    as a fraction of the font size.
rect : tuple (left, bottom, right, top), default: (0, 0, 1, 1)
    A rectangle in normalized figure coordinates into which the whole
    subplots area (including labels) will fit.

See Also
--------
.Figure.set_layout_engine
.pyplot.tight_layout
File:      /nfs/team205/kk18/miniconda3/envs/celloracle_env/lib/python3.8/site-packages/matplotlib/figure.py
Type:      method
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/1_prepare_for_multivi.html

**Contents:**
- Variables¶
- Prepare paired anndata¶
- Prepare unimodality data¶

12964 rows × 40 columns

544641 rows × 3 columns

**Examples:**

Example 1 (python):
```python
import scanpy as sc
import pandas as pd
import numpy as np
import anndata
import scipy
import os

from scipy.io import mmwrite, mmread
from scipy.sparse import csr_matrix

import sctk
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (unknown):
```unknown
-----
anndata             0.8.0
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
scipy               1.11.1
sctk                NA
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                         10.0.0
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.07.22
cffi                        1.15.1
charset_normalizer          3.2.0
comm                        0.1.3
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.7
decorator                   5.1.1
defusedxml                  0.7.1
executing                   1.2.0
fastjsonschema              NA
fqdn                        NA
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
ipykernel                   6.25.0
isoduration                 NA
jedi                        0.18.2
jinja2                      3.1.2
joblib                      1.3.1
json5                       NA
jsonpointer                 2.0
jsonschema                  4.18.4
jsonschema_specifications   NA
jupyter_events              0.6.3
jupyter_server              2.7.0
jupyterlab_server           2.24.0
kiwisolver                  1.4.4
leidenalg                   0.10.1
llvmlite                    0.40.1
markupsafe                  2.1.3
matplotlib                  3.7.2
mpl_toolkits                NA
natsort                     8.4.0
nbformat                    5.9.1
numba                       0.57.1
numpy_groupies              0.9.22
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
platformdirs                3.9.1
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pyarrow                     12.0.1
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.15.1
pyparsing                   3.0.9
pythonjsonlogger            NA
pytz                        2023.3
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
seaborn                     0.12.2
send2trash                  NA
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
stack_data                  0.6.2
statsmodels                 0.14.0
texttable                   1.6.7
threadpoolctl               3.2.0
tornado                     6.3.2
traitlets                   5.9.0
typing_extensions           NA
uri_template                NA
urllib3                     1.26.16
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.1
yaml                        6.0
zmq                         25.1.0
```

Example 5 (unknown):
```unknown
-----
IPython             8.14.0
jupyter_client      8.3.0
jupyter_core        5.3.1
jupyterlab          4.0.3
notebook            7.0.0
-----
Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0]
Linux-5.4.0-163-generic-x86_64-with-glibc2.27
-----
Session information updated at 2023-10-19 18:44
```

Example 6 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 7 (python):
```python
job_name="aCMs_AnyRegion" #  corresponds to name on input adata(s) .h5ad files. E.g. cell type or bespoke job like "aCMs_WholeSample"
```

Example 8 (unknown):
```unknown
path_to_RNA_object='/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw.h5ad' # global object
path_to_ATAC_object='/nfs/team205/heart/anndata_objects/Foetal/multiome_ATAC/ArchR/project_output/PeakMatrix/Foetal_Peaks.h5ad' # global object
```

Example 9 (unknown):
```unknown
celltype_key = 'fine_grain'
celltypes_to_analyse = ['AtrialCardiomyocytesRight','AtrialCardiomyocytesLeft']
```

Example 10 (unknown):
```unknown
region_subset=False

if region_subset:
    region_key='region'
    regions_to_analyse = ['whole sample']
```

Example 11 (unknown):
```unknown
obs_to_keep=['sangerID', 'combinedID', 'donor', 
                       'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 
                       'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts',
                       'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts',
                       'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt',
                       'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo',
                       'pct_counts_ribo', 'HB_score', 'multiplexed',
                        'batch_key','region','FACSgate','sex','week','trimester',
                     'fine_grain','mid_grain','coarse_grain',
                     'heart_or_greatvessels','cycling','S_score','G2M_score','phase','stress_score','hb1_score']
```

Example 12 (unknown):
```unknown
RNA_var_to_keep=['gene_name_scRNA-0','gene_id']
```

Example 13 (python):
```python
paired_adata_out_path=os.path.join("/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA",f"{job_name}_paired.h5ad")
paired_adata_out_path
```

Example 14 (unknown):
```unknown
'/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA/aCMs_AnyRegion_paired.h5ad'
```

Example 15 (unknown):
```unknown
multiome_only=True
```

Example 16 (python):
```python
# rna
adata_rna = sc.read_h5ad(path_to_RNA_object)

# subset cells
adata_rna = adata_rna[adata_rna.obs[celltype_key].isin(celltypes_to_analyse)]

# subset regions
if region_subset:
    adata_rna = adata_rna[adata_rna.obs[region_key].isin(regions_to_analyse)]

# add modality to .var
adata_rna.var['modality']='Gene Expression'

adata_rna
```

Example 17 (python):
```python
/tmp/ipykernel_211566/3076047041.py:12: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata_rna.var['modality']='Gene Expression'
```

Example 18 (unknown):
```unknown
AnnData object with n_obs × n_vars = 17302 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id', 'modality'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 19 (python):
```python
pd.crosstab(adata_rna.obs[celltype_key],adata_rna.obs['region'])
```

Example 20 (python):
```python
pd.crosstab(adata_rna.obs[celltype_key],adata_rna.obs['kit_10x'])
```

Example 21 (python):
```python
pd.crosstab(adata_rna.obs[celltype_key],adata_rna.obs['cycling'])
```

Example 22 (python):
```python
# atac
# ie only Multiome data
adata_atac = sc.read_h5ad(path_to_ATAC_object)

# subset cells
adata_atac = adata_atac[adata_atac.obs[celltype_key].isin(celltypes_to_analyse)]
print(adata_atac.obs[celltype_key].value_counts())

# subset regions
if region_subset:
    adata_atac = adata_atac[adata_atac.obs[region_key].isin(regions_to_analyse)]

# add modality to .var
adata_atac.var['modality']='Peaks'

adata_atac
```

Example 23 (unknown):
```unknown
AtrialCardiomyocytesRight    8187
AtrialCardiomyocytesLeft     4777
Name: fine_grain, dtype: int64
```

Example 24 (python):
```python
/tmp/ipykernel_211566/3253878154.py:14: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata_atac.var['modality']='Peaks'
```

Example 25 (unknown):
```unknown
AnnData object with n_obs × n_vars = 12964 × 508040
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels'
    var: 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N', 'modality'
```

Example 26 (python):
```python
pd.crosstab(adata_atac.obs[celltype_key],adata_atac.obs['region'])
```

Example 27 (python):
```python
# shared barcodes, ie Multiome data (not all, since MultiomeRNA has a bit more data than MultiomeATAC)
barcodes_inter=list(set(adata_rna.obs_names).intersection(adata_atac.obs_names))
len(barcodes_inter)

# take shared cells
multiome_rna=adata_rna[barcodes_inter,:]
multiome_atac=adata_atac[barcodes_inter,:]

# concatenate
adata_paired=anndata.AnnData(
    X=scipy.sparse.hstack([multiome_rna.X,multiome_atac.X]).tocsr(), # concatenate sparse matrix and convert to Compressed Sparse Row format
    obs=multiome_rna.obs[obs_to_keep],
    var=pd.concat([multiome_rna.var[['modality']+RNA_var_to_keep], \
                   multiome_atac.var[['modality']]
                  ],axis=0)
)
adata_paired
```

Example 28 (unknown):
```unknown
AnnData object with n_obs × n_vars = 12964 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
```

Example 29 (python):
```python
# modify modality columns in .obs
adata_paired.obs['modality'] = 'Multiome'
adata_paired.obs['modality'].value_counts()
```

Example 30 (unknown):
```unknown
Multiome    12964
Name: modality, dtype: int64
```

Example 31 (python):
```python
adata_paired.obs
```

Example 32 (python):
```python
adata_paired.var
```

Example 33 (python):
```python
adata_paired.X.data[:10]
```

Example 34 (unknown):
```unknown
array([1., 1., 1., 1., 1., 1., 2., 1., 2., 3.], dtype=float32)
```

Example 35 (python):
```python
# save
adata_paired.write(paired_adata_out_path)
print(f"saved to {paired_adata_out_path}")
```

Example 36 (unknown):
```unknown
saved to /lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA/aCMs_AnyRegion_paired.h5ad
```

Example 37 (python):
```python
# If there is an object of different modality e.g. scRNAseq then make that too
if multiome_only:
    print("only making a single multiome object...")
else:
    print("making unimodality data too")
    modality_col="scRNA" # define what you want to go into modality column
    unimodality_adata_out_path=os.path.join("/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA",f"{job_name}_unimodality.h5ad")

    # select single-cell data (3p and 5p)
    adata_unimodality = adata_rna[adata_rna.obs['kit_10x'].isin(['3prime-v3','5prime-v2'])]
    adata_unimodality.obs['kit_10x'].value_counts()

    # make adata
    adata_unimodality=anndata.AnnData(
        X=adata_scrna.X, # concatenate sparse matrix and convert to Compressed Sparse Row format
        obs=adata_scrna.obs[obs_to_keep],
        var=adata_scrna.var[['modality']+RNA_var_to_keep]
    )

    # modify modality columns in .obs
    adata_unimodality.obs['modality'] = modality_col
    adata_unimodality.obs['modality'].value_counts()

    adata_unimodality.write(unimodality_adata_out_path)
    print(f"saved to {unimodality_adata_out_path}")
```

Example 38 (unknown):
```unknown
only making a single multiome object...
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/Visium_mapping/c2l_pt1_reference.html

**Contents:**
- Loading packages¶
- Functions¶
- Loading scRNA-seq reference data¶
- Make sure .var_names are ENSEMBL IDs¶
- Make reference signatures¶

297473 rows × 1 columns

**Examples:**

Example 1 (unknown):
```unknown
run = "20Mar2024_fetal_OCT"
```

Example 2 (unknown):
```unknown
celltype_key = 'fine_grain_modified'
```

Example 3 (python):
```python
# create paths and names to results folders for reference regression and cell2location models
import os
results_folder = os.path.join(os.getcwd(),"analysis/Visium/cell2location_results")
ref_run_name = f'{results_folder}/reference_signatures/{run}'
run_name = f'{results_folder}/cell2location_map/{run}'

os.makedirs(ref_run_name, exist_ok=True)
os.makedirs(run_name, exist_ok=True)
```

Example 4 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 5 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Global seed set to 0
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 6 (python):
```python
import warnings
# Suppress UserWarnings
warnings.filterwarnings('ignore', category=UserWarning)
```

Example 7 (python):
```python
import session_info
session_info.show()
```

Example 8 (python):
```python
-----
anndata             0.8.0
cell2location       0.1.3
matplotlib          3.7.2
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
scvi                0.20.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 9 (unknown):
```unknown
PIL                         10.0.0
absl                        NA
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.07.22
cffi                        1.15.1
charset_normalizer          3.2.0
chex                        0.1.7
comm                        0.1.3
contextlib2                 NA
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.7
decorator                   5.1.1
defusedxml                  0.7.1
docrep                      0.3.2
etils                       1.4.0
executing                   1.2.0
fastjsonschema              NA
flax                        0.7.0
fqdn                        NA
fsspec                      2023.6.0
graphviz                    0.20.1
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_resources         NA
ipykernel                   6.25.0
isoduration                 NA
jax                         0.4.14
jaxlib                      0.4.14
jedi                        0.18.2
jinja2                      3.1.2
joblib                      1.3.1
json5                       NA
jsonpointer                 2.0
jsonschema                  4.18.4
jsonschema_specifications   NA
jupyter_events              0.6.3
jupyter_server              2.7.0
jupyterlab_server           2.24.0
kiwisolver                  1.4.4
leidenalg                   0.10.1
lightning_fabric            1.9.5
lightning_utilities         0.9.0
llvmlite                    0.40.1
markupsafe                  2.1.3
ml_collections              NA
ml_dtypes                   0.2.0
mpl_toolkits                NA
mpmath                      1.3.0
msgpack                     1.0.5
mudata                      0.2.3
multipledispatch            0.6.0
natsort                     8.4.0
nbformat                    5.9.1
numba                       0.57.1
numpyro                     0.12.1
nvfuser                     NA
opt_einsum                  v3.3.0
optax                       0.1.7
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
platformdirs                3.9.1
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pyarrow                     12.0.1
pycparser                   2.21
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.15.1
pynndescent                 0.5.10
pyparsing                   3.0.9
pyro                        1.8.6
pythonjsonlogger            NA
pytorch_lightning           1.9.5
pytz                        2023.3
referencing                 NA
regex                       2.5.129
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rich                        NA
rpds                        NA
scipy                       1.11.1
send2trash                  NA
setuptools                  68.0.0
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
sparse                      0.14.0
stack_data                  0.6.2
statsmodels                 0.14.0
sympy                       1.12
texttable                   1.6.7
threadpoolctl               3.2.0
toolz                       0.12.0
torch                       2.0.1+cu117
torchmetrics                1.0.1
tornado                     6.3.2
tqdm                        4.65.0
traitlets                   5.9.0
tree                        0.1.8
typing_extensions           NA
umap                        0.5.3
uri_template                NA
urllib3                     1.26.16
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.1
yaml                        6.0
zmq                         25.1.0
```

Example 10 (unknown):
```unknown
-----
IPython             8.14.0
jupyter_client      8.3.0
jupyter_core        5.3.1
jupyterlab          4.0.3
notebook            7.0.0
-----
Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0]
Linux-5.15.0-88-generic-x86_64-with-glibc2.35
-----
Session information updated at 2023-11-21 11:02
```

Example 11 (python):
```python
# Importing necessary libraries
from cell2location.utils.filtering import filter_genes
from cell2location.models import RegressionModel

# Function to make c2l refernce
def c2l_make_reference(adata, subset_value,max_epochs=250):
    print(f'##### {subset_value} #####')
    print(adata.obs[celltype_key].value_counts())

    print(f'filtering features...')
    selected = filter_genes(adata, cell_count_cutoff=5, cell_percentage_cutoff2=0.03, nonz_mean_cutoff=1.12)
    adata = adata[:, selected].copy()

    print(f'removing cell types which occur very seldom...')
    counts = adata.obs[celltype_key].value_counts()
    adata = adata[adata.obs[celltype_key].isin(list(counts.keys()[counts > 2]))]

    print(f'setting up model...')
    RegressionModel.setup_anndata(adata=adata, batch_key='batch_key', labels_key=celltype_key)
    mod = RegressionModel(adata)
    mod.view_anndata_setup()
    
    print(f'training model for {max_epochs} epochs...')
    mod.train(max_epochs=max_epochs, batch_size=2500, train_size=1, lr=0.002, use_gpu=True)
    
    print(f'view history...')
    mod.plot_history(20)
    
    print(f'extract posteriors...')
    adata = mod.export_posterior(adata, sample_kwargs={'num_samples': 1000, 'batch_size': 2500, 'use_gpu': True})
    print(f'saving model to {ref_run_name}/{subset_value}...')
    mod.save(f"{ref_run_name}/{subset_value}/", overwrite=True)
    print(f'saving reference to {ref_run_name}/{subset_value}...')
    adata.write(f"{ref_run_name}/{subset_value}/sc.h5ad")
    print(f'plotting QC...')
    mod.plot_QC()
    
    if 'means_per_cluster_mu_fg' in adata.varm.keys():
        inf_aver = adata.varm['means_per_cluster_mu_fg'][[f'means_per_cluster_mu_fg_{i}' for i in adata.uns['mod']['factor_names']]].copy()
    else:
        inf_aver = adata.var[[f'means_per_cluster_mu_fg_{i}' for i in adata.uns['mod']['factor_names']]].copy()
    inf_aver.columns = adata.uns['mod']['factor_names']
    inf_aver.to_csv(f"{ref_run_name}/{subset_value}/inf_aver.csv")
    
    del adata, mod, inf_aver, selected, counts
```

Example 12 (python):
```python
# Read scRNA data
scRNA_path='/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw.h5ad'
adata = sc.read(scRNA_path)
adata
```

Example 13 (unknown):
```unknown
AnnData object with n_obs × n_vars = 297473 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'mid_grain_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 14 (python):
```python
adata.X.data[:10]
```

Example 15 (unknown):
```unknown
array([ 1.,  1.,  5.,  5.,  8.,  4., 13.,  4.,  3.,  1.], dtype=float32)
```

Example 16 (python):
```python
adata.layers["counts"]=adata.X.copy() # counts to .X
```

Example 17 (python):
```python
adata.X.data[:10]
```

Example 18 (unknown):
```unknown
array([ 1.,  1.,  5.,  5.,  8.,  4., 13.,  4.,  3.,  1.], dtype=float32)
```

Example 19 (python):
```python
# Import custom annotation
fine_grain_modified=pd.read_csv("fine_grain_modified.csv", index_col=0)
fine_grain_modified
```

Example 20 (python):
```python
adata.obs=adata.obs.merge(fine_grain_modified,left_index=True, right_index=True)
adata.obs.head(3)
```

Example 21 (python):
```python
set(adata.obs[celltype_key])
```

Example 22 (unknown):
```unknown
{'AtrialCardiomyocytesCycling',
 'AtrialCardiomyocytesImmature',
 'AtrialCardiomyocytesLeft',
 'AtrialCardiomyocytesRight',
 'AtrioventricularNodeCardiomyocytes',
 'BCells',
 'BCellsMS4A1pos',
 'ChromaffinCells',
 'CoronaryArterialEndothelialCells',
 'CoronaryCapillaryEndothelialCells',
 'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'CoronaryVenousEndothelialCells',
 'CoronaryVesselAdventitialFibroblasts',
 'DendriticCellsMature',
 'DendriticCellsType1',
 'DuctusArteriosusSmoothMuscleCells',
 'EndocardialCells',
 'EndocardialCushionCells',
 'EpicardiumDerivedCells',
 'GreatVesselAdventitialFibroblasts',
 'GreatVesselArterialEndothelialCells',
 'GreatVesselSmoothMuscleCells',
 'GreatVesselVenousEndothelialCells',
 'InnateLymphoidCells',
 'LymphNodeFibroblasticReticularCells',
 'LymphaticEndothelialCells',
 'MacrophagesATF3pos',
 'MacrophagesCX3CR1pos',
 'MacrophagesLYVE1pos',
 'MacrophagesTIMD4pos',
 'MastCells',
 'Megakaryocytes',
 'MesothelialEpicardialCells',
 'MonocyteDerivedCells',
 'Monocytes',
 'MonocytesMPOpos',
 'MyocardialInterstitialFibroblasts1',
 'MyocardialInterstitialFibroblasts2',
 'Myofibroblasts',
 'NaturalKillerCells',
 'NeuronPrecursors',
 'ParaympatheticNeurons',
 'PericardialCells1',
 'PericardialCells2',
 'PericardialCells3',
 'PlasmacytoidDendriticCells',
 'ProBCells',
 'SchwannCellPrecursors',
 'SchwannCells',
 'SinoatrialNodeCardiomyocytes',
 'SympatheticNeurons',
 'TCellsCD4pos',
 'TCellsCD8pos',
 'TregsCD4pos',
 'ValveEndothelialCells',
 'ValveInterstitialCells',
 'VentricularCardiomyocytesCycling',
 'VentricularCardiomyocytesLeftCompact',
 'VentricularCardiomyocytesLeftTrabeculated',
 'VentricularCardiomyocytesRightCompact',
 'VentricularCardiomyocytesRightTrabeculated',
 'VentricularConductionSystemDistal',
 'VentricularConductionSystemProximal'}
```

Example 23 (python):
```python
adata.var.head(3)
```

Example 24 (python):
```python
adata.var=adata.var.set_index("gene_id")
```

Example 25 (python):
```python
adata.var.head(3)
```

Example 26 (python):
```python
adata.obs['batch_key'].value_counts()
```

Example 27 (unknown):
```unknown
Hst41_Nuclei_Multiome-v1    104878
Hst33_Nuclei_Multiome-v1     38234
Hst39_Nuclei_Multiome-v1     25386
C99_Cell_5prime-v2           17547
C92_Nuclei_Multiome-v1       13770
C94_Cell_5prime-v2           12985
Hst40_Nuclei_Multiome-v1      9708
C97_Cell_5prime-v2            9702
BRC2251_Cell_3prime-v3        9121
BRC2256_Cell_3prime-v3        7948
Hst36_Nuclei_Multiome-v1      7268
BRC2252_Cell_3prime-v3        6854
C86_Cell_5prime-v2            6376
BRC2260_Cell_3prime-v3        6004
BRC2263_Cell_3prime-v3        5662
BRC2262_Cell_3prime-v3        4129
C87_Nuclei_Multiome-v1        3907
C98_Nuclei_Multiome-v1        2660
C104_Nuclei_Multiome-v1       2490
C85_Nuclei_Multiome-v1        2436
C82_Nuclei_Multiome-v1         408
Name: batch_key, dtype: int64
```

Example 28 (python):
```python
set(adata.obs['batch_key'])
```

Example 29 (unknown):
```unknown
{'BRC2251_Cell_3prime-v3',
 'BRC2252_Cell_3prime-v3',
 'BRC2256_Cell_3prime-v3',
 'BRC2260_Cell_3prime-v3',
 'BRC2262_Cell_3prime-v3',
 'BRC2263_Cell_3prime-v3',
 'C104_Nuclei_Multiome-v1',
 'C82_Nuclei_Multiome-v1',
 'C85_Nuclei_Multiome-v1',
 'C86_Cell_5prime-v2',
 'C87_Nuclei_Multiome-v1',
 'C92_Nuclei_Multiome-v1',
 'C94_Cell_5prime-v2',
 'C97_Cell_5prime-v2',
 'C98_Nuclei_Multiome-v1',
 'C99_Cell_5prime-v2',
 'Hst33_Nuclei_Multiome-v1',
 'Hst36_Nuclei_Multiome-v1',
 'Hst39_Nuclei_Multiome-v1',
 'Hst40_Nuclei_Multiome-v1',
 'Hst41_Nuclei_Multiome-v1'}
```

Example 30 (unknown):
```unknown
subset=False
```

Example 31 (python):
```python
%%time
if subset:
    subset_values=adata.obs[sc_subset_key].unique().tolist()
    for value in subset_values:
        adata_ref_subset = adata[adata.obs[sc_subset_key].isin([value])].copy()
        c2l_make_reference(adata_ref_subset, value)
else:
    adata_ref = adata.copy()
    c2l_make_reference(adata_ref, "all_data")
```

Example 32 (unknown):
```unknown
##### LA #####
AtrialCardiomyocytesLeft                  19443
MP_LYVE1+                                  6955
Pericyte_1                                 5845
EC_capillary                               5764
SAN_FB                                     4311
                                          ...  
valve_FB_MV2                                  5
VentricularCardiomyocytesLeftStressed         2
valve_FB_CS                                   1
VentricularCardiomyocytesRight                1
VentricularCardiomyocytesRightStressed        1
Name: cell_state, Length: 70, dtype: int64
filtering features...
```

Example 33 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pandas/core/arraylike.py:402: RuntimeWarning: divide by zero encountered in log10
  result = getattr(ufunc, method)(*inputs, **kwargs)
```

Example 34 (unknown):
```unknown
removing cell types which occur very seldom...
setting up model...
```

Example 35 (unknown):
```unknown
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
```

Example 36 (unknown):
```unknown
Anndata setup with scvi-tools version 0.20.3.
```

Example 37 (unknown):
```unknown
Setup via `RegressionModel.setup_anndata` with arguments:
```

Example 38 (unknown):
```unknown
{
│   'layer': None,
│   'batch_key': 'batch_key',
│   'labels_key': 'cell_state',
│   'categorical_covariate_keys': None,
│   'continuous_covariate_keys': None
}
```

Example 39 (unknown):
```unknown
Summary Statistics         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃     Summary Stat Key     ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│         n_batch          │  30   │
│         n_cells          │ 89744 │
│ n_extra_categorical_covs │   0   │
│ n_extra_continuous_covs  │   0   │
│         n_labels         │  66   │
│          n_vars          │ 13571 │
└──────────────────────────┴───────┘
```

Example 40 (python):
```python
Data Registry                
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Registry Key ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│      X       │          adata.X          │
│    batch     │ adata.obs['_scvi_batch']  │
│    ind_x     │   adata.obs['_indices']   │
│    labels    │ adata.obs['_scvi_labels'] │
└──────────────┴───────────────────────────┘
```

Example 41 (python):
```python
batch State Registry                           
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃    Source Location     ┃       Categories        ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['batch_key'] │ A61_Nuclei_Multiome-v1  │          0          │
│                        │ AH5_Nuclei_Multiome-v1  │          1          │
│                        │ AV1_Nuclei_Multiome-v1  │          2          │
│                        │ AV2_Nuclei_Multiome-v1  │          3          │
│                        │ AV10_Nuclei_Multiome-v1 │          4          │
│                        │ AV11_Nuclei_Multiome-v1 │          5          │
│                        │ AV13_Nuclei_Multiome-v1 │          6          │
│                        │ AV14_Nuclei_Multiome-v1 │          7          │
│                        │   D1_Nuclei_3prime-v2   │          8          │
│                        │   D2_Nuclei_3prime-v2   │          9          │
│                        │    D3_Cell_3prime-v2    │         10          │
│                        │   D3_Nuclei_3prime-v2   │         11          │
│                        │    D4_Cell_3prime-v2    │         12          │
│                        │   D4_Nuclei_3prime-v2   │         13          │
│                        │    D5_Cell_3prime-v2    │         14          │
│                        │   D5_Nuclei_3prime-v2   │         15          │
│                        │    D6_Cell_3prime-v2    │         16          │
│                        │    D6_Cell_3prime-v3    │         17          │
│                        │   D6_Nuclei_3prime-v2   │         18          │
│                        │    D7_Cell_3prime-v2    │         19          │
│                        │   D7_Nuclei_3prime-v2   │         20          │
│                        │  D8_Nuclei_Multiome-v1  │         21          │
│                        │   D11_Cell_3prime-v3    │         22          │
│                        │  D11_Nuclei_3prime-v3   │         23          │
│                        │   H2_Nuclei_3prime-v3   │         24          │
│                        │   H3_Nuclei_3prime-v3   │         25          │
│                        │   H4_Nuclei_3prime-v3   │         26          │
│                        │   H5_Nuclei_3prime-v3   │         27          │
│                        │   H6_Nuclei_3prime-v3   │         28          │
│                        │   H7_Nuclei_3prime-v3   │         29          │
└────────────────────────┴─────────────────────────┴─────────────────────┘
```

Example 42 (python):
```python
labels State Registry                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃     Source Location     ┃            Categories             ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['cell_state'] │            Adipocytes             │          0          │
│                         │       AppendageNeuralCells        │          1          │
│                         │       Artifact mesothelial        │          2          │
│                         │          Atrial pericyte          │          3          │
│                         │     AtrialCardiomyocytesLeft      │          4          │
│                         │     AtrialCardiomyocytesRight     │          5          │
│                         │      AtrialConductionSystem       │          6          │
│                         │                 B                 │          7          │
│                         │             B_plasma              │          8          │
│                         │               CD4+T               │          9          │
│                         │            CD4+T_naive            │         10          │
│                         │             CD4+T_reg             │         11          │
│                         │               CD8+T               │         12          │
│                         │     CD55+ FGFR1+ mesothelial      │         13          │
│                         │        CRIM1+ mesothelial         │         14          │
│                         │           Coronary vSMC           │         15          │
│                         │     CoronaryArteryAdipocytes      │         16          │
│                         │                DC                 │         17          │
│                         │            EC_arterial            │         18          │
│                         │           EC_capillary            │         19          │
│                         │       EC_capillary_IGFBP5+        │         20          │
│                         │        EC_capillary_VEGFA+        │         21          │
│                         │     EC_capillary_VEGFA+NOVA1+     │         22          │
│                         │        EC_capillary_immune        │         23          │
│                         │          EC_endocardial           │         24          │
│                         │             EC_venous             │         25          │
│                         │         EClikeNeuralCells         │         26          │
│                         │         Great vessel vSMC         │         27          │
│                         │     LymphaticEndothelialCells     │         28          │
│                         │             MP_CXCL8+             │         29          │
│                         │             MP_FABP5+             │         30          │
│                         │             MP_LYVE1+             │         31          │
│                         │          MP_LYVE1+AGBL4+          │         32          │
│                         │            MP_cycling             │         33          │
│                         │             MastCells             │         34          │
│                         │        Microvascular SMC_1        │         35          │
│                         │        Microvascular SMC_2        │         36          │
│                         │               Mo/MP               │         37          │
│                         │               Mono                │         38          │
│                         │             NK_CD16hi             │         39          │
│                         │             NK_CD56hi             │         40          │
│                         │            NeuralCells            │         41          │
│                         │         NodalNeuralCells          │         42          │
│                         │          PacemakerCells           │         43          │
│                         │            Pericyte_1             │         44          │
│                         │            Pericyte_2             │         45          │
│                         │            Pericyte_3             │         46          │
│                         │        Proliferating vSMC         │         47          │
│                         │      PulmonaryVeinAdipocytes      │         48          │
│                         │              SAN_FB               │         49          │
│                         │     SOD2+ TIMP1+ mesothelial      │         50          │
│                         │   SinoatrialNodeCardiomyocytes    │         51          │
│                         │   VentricularCardiomyocytesLeft   │         52          │
│                         │ VentricularConductionSystemDistal │         53          │
│                         │         VesselNeuralCells         │         54          │
│                         │               aFB1                │         55          │
│                         │               aFB4                │         56          │
│                         │             doublets              │         57          │
│                         │               vFB1                │         58          │
│                         │          vFB2_activated           │         59          │
│                         │              vSMC_1               │         60          │
│                         │              vSMC_2               │         61          │
│                         │            valve_FB_AV            │         62          │
│                         │           valve_FB_MV1            │         63          │
│                         │           valve_FB_MV2            │         64          │
│                         │            vascular_FB            │         65          │
└─────────────────────────┴───────────────────────────────────┴─────────────────────┘
```

Example 43 (unknown):
```unknown
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
```

Example 44 (unknown):
```unknown
training model for 250 epochs...
```

Example 45 (unknown):
```unknown
You are using a CUDA device ('NVIDIA A100-SXM4-80GB') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
```

Example 46 (python):
```python
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File <timed exec>:5

Cell In[17], line 24, in c2l_make_reference(adata, subset_value, max_epochs)
     21 mod.view_anndata_setup()
     23 print(f'training model for {max_epochs} epochs...')
---> 24 mod.train(max_epochs=max_epochs, batch_size=2500, train_size=1, lr=0.002, use_gpu=True)
     26 print(f'view history...')
     27 mod.plot_history(20)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/cell2location/models/reference/_reference_model.py:157, in RegressionModel.train(self, max_epochs, batch_size, train_size, lr, **kwargs)
    154 kwargs["train_size"] = train_size
    155 kwargs["lr"] = lr
--> 157 super().train(**kwargs)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scvi/model/base/_pyromixin.py:172, in PyroSviTrainMixin.train(self, max_epochs, use_gpu, train_size, validation_size, batch_size, early_stopping, lr, training_plan, plan_kwargs, **trainer_kwargs)
    162 trainer_kwargs["callbacks"].append(PyroJitGuideWarmup())
    164 runner = self._train_runner_cls(
    165     self,
    166     training_plan=training_plan,
   (...)
    170     **trainer_kwargs,
    171 )
--> 172 return runner()

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scvi/train/_trainrunner.py:82, in TrainRunner.__call__(self)
     79 if hasattr(self.data_splitter, "n_val"):
     80     self.training_plan.n_obs_validation = self.data_splitter.n_val
---> 82 self.trainer.fit(self.training_plan, self.data_splitter)
     83 self._update_history()
     85 # data splitter only gets these attrs after fit

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scvi/train/_trainer.py:193, in Trainer.fit(self, *args, **kwargs)
    187 if isinstance(args[0], PyroTrainingPlan):
    188     warnings.filterwarnings(
    189         action="ignore",
    190         category=UserWarning,
    191         message="`LightningModule.configure_optimizers` returned `None`",
    192     )
--> 193 super().fit(*args, **kwargs)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py:608, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    606 model = self._maybe_unwrap_optimized(model)
    607 self.strategy._lightning_module = model
--> 608 call._call_and_handle_interrupt(
    609     self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    610 )

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py:38, in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
     36         return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
     37     else:
---> 38         return trainer_fn(*args, **kwargs)
     40 except _TunerExitException:
     41     trainer._call_teardown_hook()

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py:650, in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    643 ckpt_path = ckpt_path or self.resume_from_checkpoint
    644 self._ckpt_path = self._checkpoint_connector._set_ckpt_path(
    645     self.state.fn,
    646     ckpt_path,  # type: ignore[arg-type]
    647     model_provided=True,
    648     model_connected=self.lightning_module is not None,
    649 )
--> 650 self._run(model, ckpt_path=self.ckpt_path)
    652 assert self.state.stopped
    653 self.training = False

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py:1093, in Trainer._run(self, model, ckpt_path)
   1090 self._logger_connector.reset_metrics()
   1092 # strategy will configure model and move it to the device
-> 1093 self.strategy.setup(self)
   1095 # hook
   1096 if self.state.fn == TrainerFn.FITTING:

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/strategies/single_device.py:73, in SingleDeviceStrategy.setup(self, trainer)
     72 def setup(self, trainer: pl.Trainer) -> None:
---> 73     self.model_to_device()
     74     super().setup(trainer)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/strategies/single_device.py:70, in SingleDeviceStrategy.model_to_device(self)
     68 def model_to_device(self) -> None:
     69     assert self.model is not None, "self.model must be set before self.model.to()"
---> 70     self.model.to(self.root_device)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/lightning_fabric/utilities/device_dtype_mixin.py:54, in _DeviceDtypeModuleMixin.to(self, *args, **kwargs)
     52 device, dtype = torch._C._nn._parse_to(*args, **kwargs)[:2]
     53 self.__update_properties(device=device, dtype=dtype)
---> 54 return super().to(*args, **kwargs)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/modules/module.py:1145, in Module.to(self, *args, **kwargs)
   1141         return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
   1142                     non_blocking, memory_format=convert_to_format)
   1143     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
-> 1145 return self._apply(convert)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/modules/module.py:797, in Module._apply(self, fn)
    795 def _apply(self, fn):
    796     for module in self.children():
--> 797         module._apply(fn)
    799     def compute_should_use_set_data(tensor, tensor_applied):
    800         if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):
    801             # If the new tensor has compatible tensor type as the existing tensor,
    802             # the current behavior is to change the tensor in-place using `.data =`,
   (...)
    807             # global flag to let the user control whether they want the future
    808             # behavior of overwriting the existing tensor or not.

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/modules/module.py:797, in Module._apply(self, fn)
    795 def _apply(self, fn):
    796     for module in self.children():
--> 797         module._apply(fn)
    799     def compute_should_use_set_data(tensor, tensor_applied):
    800         if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):
    801             # If the new tensor has compatible tensor type as the existing tensor,
    802             # the current behavior is to change the tensor in-place using `.data =`,
   (...)
    807             # global flag to let the user control whether they want the future
    808             # behavior of overwriting the existing tensor or not.

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/modules/module.py:844, in Module._apply(self, fn)
    842 for key, buf in self._buffers.items():
    843     if buf is not None:
--> 844         self._buffers[key] = fn(buf)
    846 return self

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/modules/module.py:1143, in Module.to.<locals>.convert(t)
   1140 if convert_to_format is not None and t.dim() in (4, 5):
   1141     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
   1142                 non_blocking, memory_format=convert_to_format)
-> 1143 return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/TransmuralAxis/tm-axis_cell2loc_compact-trab_transition.html

**Contents:**
- Import modules¶
- Functions¶
- Read in adata¶
- Add axis¶
- Plot transmural axis¶
- Concatenate OCT and FFPE object¶
- Plot cells along the axis and time¶
- Cell type transition along transmural axis¶
- Plot compact and trabeculated CM for each age¶

**Examples:**

Example 1 (python):
```python
import os
import numpy as np
import scanpy as sc
import anndata
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 
import seaborn as sns

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter("ignore", UserWarning)
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
anndata             0.9.2
matplotlib          3.7.2
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                         10.0.0
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.11.17
cffi                        1.15.1
charset_normalizer          3.2.0
colorama                    0.4.6
comm                        0.1.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.8
decorator                   5.1.1
defusedxml                  0.7.1
executing                   1.2.0
fastjsonschema              NA
fqdn                        NA
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_metadata          NA
importlib_resources         NA
ipykernel                   6.25.1
isoduration                 NA
jedi                        0.19.0
jinja2                      3.1.2
joblib                      1.3.2
json5                       NA
jsonpointer                 2.0
jsonschema                  4.19.0
jsonschema_specifications   NA
jupyter_events              0.7.0
jupyter_server              2.7.1
jupyterlab_server           2.24.0
kiwisolver                  1.4.5
leidenalg                   0.10.1
llvmlite                    0.40.1
markupsafe                  2.1.3
mpl_toolkits                NA
natsort                     8.4.0
nbformat                    5.9.2
numba                       0.57.1
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.16.1
pyparsing                   3.0.9
pythonjsonlogger            NA
pytz                        2023.3
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
ruamel                      NA
scipy                       1.10.1
send2trash                  NA
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
sphinxcontrib               NA
stack_data                  0.6.2
statsmodels                 0.14.0
texttable                   1.6.7
threadpoolctl               3.2.0
tornado                     6.3.3
traitlets                   5.9.0
typing_extensions           NA
uri_template                NA
urllib3                     2.0.4
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.2
yaml                        6.0.1
zipp                        NA
zmq                         25.1.1
zstandard                   0.19.0
```

Example 5 (unknown):
```unknown
-----
IPython             8.12.0
jupyter_client      8.3.0
jupyter_core        4.12.0
jupyterlab          4.0.5
notebook            7.0.2
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-5.4.0-169-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-03-20 22:41
```

Example 6 (python):
```python
import importlib.util
import sys

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/tissueaxis.py")
tissueaxis = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = tissueaxis
spec.loader.exec_module(tissueaxis)
```

Example 7 (python):
```python
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 8 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 9 (python):
```python
# This function selects the data for multiple slides from the spatial anndata object.
# Modified `select_slide` in cell2loc for multiple slides

# @param adata: anndata object with multiple spatial experiments
# @param s: list of selected slides
# @param s_col: column in adata.obs listing experiment name for each location

def select_multislide(adata, s, s_col='sample'):

    slides = adata[adata.obs[s_col].isin(s), :]
    s_keys = list(slides.uns['spatial'].keys())
    
    uns_dict = {}
    for ss in s:
        s_spatial=np.array(s_keys)[[ss in k for k in s_keys]][0]
        uns_dict[s_spatial]=slides.uns['spatial'][s_spatial]
    
    slides.uns['spatial'] = uns_dict

    return slides
```

Example 10 (python):
```python
adata_ffpe = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-FFPE_foetal_raw.h5ad')
print(adata_ffpe.shape)
adata_oct_healthy = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT_foetal_raw.h5ad')
print(adata_oct_healthy.shape)
adata_oct_healthy_LV = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT-LV_foetal_raw.h5ad')
print(adata_oct_healthy_LV.shape)
adata_oct_t21 = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT-T21_foetal_raw.h5ad')
print(adata_oct_t21.shape)
```

Example 11 (unknown):
```unknown
(19973, 18085)
(14261, 33538)
(6767, 33538)
(11238, 33538)
```

Example 12 (python):
```python
# concatenate oct adata
adata_oct_healthy.obs['status'] = 'Healthy'
adata_oct_healthy_LV.obs['status'] = 'Healthy'
adata_oct_t21.obs['status'] = 'Trisomy21'

adata_oct = adata_oct_healthy.concatenate(
    [adata_oct_healthy_LV,adata_oct_t21],
    batch_key=None,
    index_unique=None,
    join="inner"
)

adata_oct.obs['status'].value_counts()
```

Example 13 (unknown):
```unknown
Healthy      21028
Trisomy21    11238
Name: status, dtype: int64
```

Example 14 (python):
```python
# log-normalise
sc.pp.normalize_total(adata_oct, target_sum=1e4)
sc.pp.log1p(adata_oct)

sc.pp.normalize_total(adata_ffpe, target_sum=1e4)
sc.pp.log1p(adata_ffpe)

# reset var_names
adata_oct.var.set_index('SYMBOL',inplace=True)
adata_ffpe.var.set_index('SYMBOL',inplace=True)
```

Example 15 (python):
```python
# get all celltypes
# only the LV cell types will be appeared since concatenated with LV object
all_celltypes = adata_oct.obsm['q05_cell_abundance_w_sf'].columns
all_celltypes = [x.replace('q05cell_abundance_w_sf_','') for x in all_celltypes]
print(len(all_celltypes))
print(all_celltypes[:5])
```

Example 16 (unknown):
```unknown
41
['VentricularCardiomyocytesLeftCompact', 'VentricularCardiomyocytesLeftTrabeculated', 'VentricularConductionSystemProximal', 'VentricularConductionSystemDistal', 'CoronaryVesselAdventitialFibroblasts']
```

Example 17 (python):
```python
# subset cell types in ffpe
adata_ffpe.obsm['q05_cell_abundance_w_sf'] = adata_ffpe.obsm['q05_cell_abundance_w_sf'][[f'q05cell_abundance_w_sf_{x}' for x in all_celltypes]]

# modify cell type names
celltype_rename = utils.read_pkl('/nfs/team205/heart/anndata_objects/Foetal/fine_grain_rename_dictionary.pkl')
adata_ffpe.obsm['q05_cell_abundance_w_sf'].columns = [x.replace(x.split('q05cell_abundance_w_sf_')[1],celltype_rename[x.split('q05cell_abundance_w_sf_')[1]]) \
                                                      for x in adata_ffpe.obsm['q05_cell_abundance_w_sf'].columns]
adata_oct.obsm['q05_cell_abundance_w_sf'].columns = [x.replace(x.split('q05cell_abundance_w_sf_')[1],celltype_rename[x.split('q05cell_abundance_w_sf_')[1]]) \
                                                      for x in adata_oct.obsm['q05_cell_abundance_w_sf'].columns]
```

Example 18 (python):
```python
# add
ffpe_samples = list(set(adata_ffpe.obs['sample']))
oct_samples = list(set(adata_oct.obs['sample']))

adata_oct.obs['transmural_axis'] = np.nan
adata_ffpe.obs['transmural_axis'] = np.nan

axis_files = os.listdir('/nfs/team205/heart/visium_axis/foetal/transmural_LV')
for file in axis_files:
    # read in
    axis = pd.read_csv(f'/nfs/team205/heart/visium_axis/foetal/transmural_LV/{file}',index_col=0)
    # add to anndata
    if file.split('.csv')[0] in oct_samples:
        adata_oct.obs.loc[axis.index,'transmural_axis'] = axis.copy()
    elif file.split('.csv')[0] in ffpe_samples:
        adata_ffpe.obs.loc[axis.index,'transmural_axis'] = axis.copy()

# select spots with axis
# adata_oct = adata_oct[adata_oct.obs['transmural_axis'].isna()==False]
# adata_ffpe = adata_ffpe[adata_ffpe.obs['transmural_axis'].isna()==False]
```

Example 19 (python):
```python
pd.crosstab(adata_oct.obs['sample'],adata_oct.obs['transmural_axis'].isna()==False)
```

Example 20 (python):
```python
pd.crosstab(adata_ffpe.obs['sample'],adata_ffpe.obs['transmural_axis'].isna()==False)
```

Example 21 (python):
```python
pd.crosstab(adata_oct.obs['sample'],adata_oct.obs['cell2loc-map_group'])
```

Example 22 (unknown):
```unknown
slide_id = 'BHF_F_Hea13295831'
```

Example 23 (python):
```python
slide = select_multislide(adata_ffpe, [slide_id])
sc.pl.spatial(slide,color=['transmural_axis'],cmap='gist_rainbow',show=False)
plt.savefig(f"{figdir}/visium_tm-axis_{slide_id}.pdf", bbox_inches="tight",dpi=300)
```

Example 24 (python):
```python
# select spots with axis
adata_oct = adata_oct[adata_oct.obs['transmural_axis'].isna()==False]
adata_ffpe = adata_ffpe[adata_ffpe.obs['transmural_axis'].isna()==False]
# for later plotting
adata_oct_healthy = adata_oct_healthy[list(set(adata_oct_healthy.obs_names).intersection(adata_oct.obs_names))]
adata_oct_healthy_LV = adata_oct_healthy_LV[list(set(adata_oct_healthy_LV.obs_names).intersection(adata_oct.obs_names))]
```

Example 25 (python):
```python
# concatenate OCT and FFPE objects
adata_ffpe.obs['cell2loc-map_group'] = '16PCW_FFPE'

adata_all = adata_oct.concatenate(
    adata_ffpe,
    batch_key=None,
    index_unique=None,
    join="inner"
)
```

Example 26 (python):
```python
pd.crosstab(adata_all.obs['sample'],adata_all.obs['cell2loc-map_group'])
```

Example 27 (python):
```python
features_to_plot = 'VentricularCardiomyocytesLeftCompact'

tissueaxis.plot_axis_split(adata_all,
                           feature_to_plot=features_to_plot,
                           feature_type='celltype',
                           splitby='cell2loc-map_group',
                           groups_order=[ '5PCW', '7PCW','13PCW', '16PCW_FFPE','20PCW'],
                           figsize=(5, 3.5),
                           cbar_pos=(0, 0.7, 0.05, 0.18),
                           save=f'{figdir}/tm-axis_{features_to_plot}_per-week.pdf'
                          )
```

Example 28 (python):
```python
features_to_plot = 'VentricularCardiomyocytesLeftTrabeculated'

tissueaxis.plot_axis_split(adata_all,
                           feature_to_plot=features_to_plot,
                           feature_type='celltype',
                           splitby='cell2loc-map_group',
                           groups_order=[ '5PCW', '7PCW','13PCW', '16PCW_FFPE','20PCW'],
                           figsize=(5, 3.5),
                           cbar_pos=(0, 0.7, 0.05, 0.18),
                           save=f'{figdir}/tm-axis_{features_to_plot}_per-week.pdf'
                          )
```

Example 29 (python):
```python
features_to_plot = 'CoronaryCapillaryEndothelialCells'

tissueaxis.plot_axis_split(adata_all,
                           feature_to_plot=features_to_plot,
                           feature_type='celltype',
                           splitby='cell2loc-map_group',
                           groups_order=[ '5PCW', '7PCW','13PCW', '16PCW_FFPE','20PCW'],
                           figsize=(5, 3.5),
                           cbar_pos=(0, 0.7, 0.05, 0.18),
                           save=f'{figdir}/tm-axis_{features_to_plot}_per-week.pdf'
                          )
```

Example 30 (python):
```python
# only euploid
tissueaxis.plot_axis(adata_all[adata_all.obs['status']=='Healthy'],
                     axis_col='transmural_axis',
                     n_bin=15,
                     feature_type='celltype',
                     correlation=False,
                     threshold_cell_abundance=0.1,
                     min_max_scale=True,
                     figsize=(3, 7),
                     save=f'{figdir}/tm-axis_OCT-FFPE_celltypes_thresh-0p1.pdf'
                    )
```

Example 31 (python):
```python
# only euploid
tissueaxis.plot_axis(adata_all[adata_all.obs['status']=='Healthy'],
                     axis_col='transmural_axis',
                     n_bin=15,
                     feature_type='celltype',
                     correlation=False,
                     threshold_cell_abundance=0.4,
                     min_max_scale=True,
                     figsize=(3, 4)
                     # save=f'{figdir}/tm-axis_OCT_celltypes.pdf'
                    )
```

Example 32 (python):
```python
# OCT, cell type transition
# only euploid
tissueaxis.plot_axis(adata_oct[adata_oct.obs['status']=='Healthy'],
                     axis_col='transmural_axis',
                     n_bin=15,
                     feature_type='celltype',
                     correlation=False,
                     threshold_cell_abundance=0.1,
                     # save=f'{figdir}/tm-axis_OCT_celltypes.pdf'
                    )
```

Example 33 (python):
```python
# FFPE, cell type transition
# only euploid
tissueaxis.plot_axis(adata_ffpe[adata_ffpe.obs['status']=='Healthy'],
                     axis_col='transmural_axis',
                     n_bin=15,
                     feature_type='celltype',
                     correlation=False,
                     threshold_cell_abundance=0.03,
                    )
```

Example 34 (python):
```python
pd.crosstab(adata_oct_healthy.obs['sample'],adata_oct_healthy.obs['cell2loc-map_group'])
```

Example 35 (python):
```python
pd.crosstab(adata_oct_healthy_LV.obs['sample'],adata_oct_healthy_LV.obs['cell2loc-map_group'])
```

Example 36 (python):
```python
slide_id = 'HCAHeartST11605167'
slide = select_multislide(adata_oct_healthy, [slide_id])
cell2loc_out = slide.obsm['q05_cell_abundance_w_sf'].copy()
cell2loc_out.columns = [x.replace('q05cell_abundance_w_sf_','') for x in cell2loc_out.columns]
slide.obs[cell2loc_out.columns] = cell2loc_out.copy()
sc.pl.spatial(slide,color=['VentricularCardiomyocytesLeftCompact','VentricularCardiomyocytesLeftTrabeculated'],show=False)
```

Example 37 (unknown):
```unknown
[<Axes: title={'center': 'VentricularCardiomyocytesLeftCompact'}, xlabel='spatial1', ylabel='spatial2'>,
 <Axes: title={'center': 'VentricularCardiomyocytesLeftTrabeculated'}, xlabel='spatial1', ylabel='spatial2'>]
```

Example 38 (python):
```python
slide_id = 'HCAHeartST11605168'
slide = select_multislide(adata_oct_healthy, [slide_id])
cell2loc_out = slide.obsm['q05_cell_abundance_w_sf'].copy()
cell2loc_out.columns = [x.replace('q05cell_abundance_w_sf_','') for x in cell2loc_out.columns]
slide.obs[cell2loc_out.columns] = cell2loc_out.copy()
sc.pl.spatial(slide,color=['VentricularCardiomyocytesLeftCompact','VentricularCardiomyocytesLeftTrabeculated'],show=False)
```

Example 39 (unknown):
```unknown
[<Axes: title={'center': 'VentricularCardiomyocytesLeftCompact'}, xlabel='spatial1', ylabel='spatial2'>,
 <Axes: title={'center': 'VentricularCardiomyocytesLeftTrabeculated'}, xlabel='spatial1', ylabel='spatial2'>]
```

Example 40 (python):
```python
slide_id = 'HCAHeartST11605165'
slide = select_multislide(adata_oct_healthy, [slide_id])
cell2loc_out = slide.obsm['q05_cell_abundance_w_sf'].copy()
cell2loc_out.columns = [x.replace('q05cell_abundance_w_sf_','') for x in cell2loc_out.columns]
slide.obs[cell2loc_out.columns] = cell2loc_out.copy()
sc.pl.spatial(slide,color=['VentricularCardiomyocytesLeftCompact','VentricularCardiomyocytesLeftTrabeculated'],show=False)
```

Example 41 (unknown):
```unknown
[<Axes: title={'center': 'VentricularCardiomyocytesLeftCompact'}, xlabel='spatial1', ylabel='spatial2'>,
 <Axes: title={'center': 'VentricularCardiomyocytesLeftTrabeculated'}, xlabel='spatial1', ylabel='spatial2'>]
```

Example 42 (python):
```python
slide_id = 'HCAHeartST10317185'
slide = select_multislide(adata_oct_healthy, [slide_id])
cell2loc_out = slide.obsm['q05_cell_abundance_w_sf'].copy()
cell2loc_out.columns = [x.replace('q05cell_abundance_w_sf_','') for x in cell2loc_out.columns]
slide.obs[cell2loc_out.columns] = cell2loc_out.copy()
sc.pl.spatial(slide,color=['VentricularCardiomyocytesLeftCompact','VentricularCardiomyocytesLeftTrabeculated'],show=False)
```

Example 43 (unknown):
```unknown
[<Axes: title={'center': 'VentricularCardiomyocytesLeftCompact'}, xlabel='spatial1', ylabel='spatial2'>,
 <Axes: title={'center': 'VentricularCardiomyocytesLeftTrabeculated'}, xlabel='spatial1', ylabel='spatial2'>]
```

Example 44 (python):
```python
slide_id = 'HCAHeartST13162330'
slide = select_multislide(adata_oct_healthy_LV, [slide_id])
cell2loc_out = slide.obsm['q05_cell_abundance_w_sf'].copy()
cell2loc_out.columns = [x.replace('q05cell_abundance_w_sf_','') for x in cell2loc_out.columns]
slide.obs[cell2loc_out.columns] = cell2loc_out.copy()
sc.pl.spatial(slide,color=['VentricularCardiomyocytesLeftCompact','VentricularCardiomyocytesLeftTrabeculated'],show=False)
```

Example 45 (unknown):
```unknown
[<Axes: title={'center': 'VentricularCardiomyocytesLeftCompact'}, xlabel='spatial1', ylabel='spatial2'>,
 <Axes: title={'center': 'VentricularCardiomyocytesLeftTrabeculated'}, xlabel='spatial1', ylabel='spatial2'>]
```

Example 46 (python):
```python
slide_id = 'HCAHeartST13162331'
slide = select_multislide(adata_oct_healthy_LV, [slide_id])
cell2loc_out = slide.obsm['q05_cell_abundance_w_sf'].copy()
cell2loc_out.columns = [x.replace('q05cell_abundance_w_sf_','') for x in cell2loc_out.columns]
slide.obs[cell2loc_out.columns] = cell2loc_out.copy()
sc.pl.spatial(slide,color=['VentricularCardiomyocytesLeftCompact','VentricularCardiomyocytesLeftTrabeculated'],show=False)
```

Example 47 (unknown):
```unknown
[<Axes: title={'center': 'VentricularCardiomyocytesLeftCompact'}, xlabel='spatial1', ylabel='spatial2'>,
 <Axes: title={'center': 'VentricularCardiomyocytesLeftTrabeculated'}, xlabel='spatial1', ylabel='spatial2'>]
```

Example 48 (python):
```python
slide_id = 'BHF_F_Hea13295831'
slide = select_multislide(adata_ffpe, [slide_id])
cell2loc_out = slide.obsm['q05_cell_abundance_w_sf'].copy()
cell2loc_out.columns = [x.replace('q05cell_abundance_w_sf_','') for x in cell2loc_out.columns]
slide.obs[cell2loc_out.columns] = cell2loc_out.copy()
sc.pl.spatial(slide,color=['VentricularCardiomyocytesLeftCompact','VentricularCardiomyocytesLeftTrabeculated'],show=False)
```

Example 49 (unknown):
```unknown
[<Axes: title={'center': 'VentricularCardiomyocytesLeftCompact'}, xlabel='spatial1', ylabel='spatial2'>,
 <Axes: title={'center': 'VentricularCardiomyocytesLeftTrabeculated'}, xlabel='spatial1', ylabel='spatial2'>]
```

Example 50 (unknown):
```unknown
[<Axes: title={'center': 'VentricularCardiomyocytesLeftCompact'}, xlabel='spatial1', ylabel='spatial2'>,
 <Axes: title={'center': 'VentricularCardiomyocytesLeftTrabeculated'}, xlabel='spatial1', ylabel='spatial2'>]
```

Example 51 (unknown):
```unknown
cell2loc_out.columns
```

Example 52 (unknown):
```unknown
Index(['VentricularCardiomyocytesLeftCompact',
       'VentricularCardiomyocytesLeftTrabeculated',
       'VentricularConductionSystemProximal',
       'VentricularConductionSystemDistal',
       'CoronaryVesselAdventitialFibroblasts',
       'MyocardialInterstitialFibroblasts', 'SubEpicardialFibroblasts',
       'Myofibroblasts', 'LymphNodeFibroblasticReticularCells',
       'CoronarySmoothMuscleCells', 'CoronaryPericytes',
       'CoronaryArterialEndothelialCells', 'CoronaryVenousEndothelialCells',
       'CoronaryCapillaryEndothelialCells', 'EndocardialCells',
       'LymphaticEndothelialCells', 'MesothelialEpicardialCells',
       'EpicardiumDerivedCells', 'NeuronPrecursors', 'ChromaffinCells',
       'SympatheticNeurons', 'ParasympatheticNeurons', 'SchwannCellPrecursors',
       'SchwannCells', 'MonocytesMPOpos', 'Monocytes', 'MonocyteDerivedCells',
       'MacrophagesCX3CR1pos', 'MacrophagesTIMD4pos', 'MacrophagesLYVE1pos',
       'MacrophagesATF3pos', 'DendriticCellsType1', 'DendriticCellsMature',
       'PlasmacytoidDendriticCells', 'MastCells', 'TCellsCD4pos',
       'TCellsCD8pos', 'TregsCD4pos', 'BCellsMS4A1pos', 'NaturalKillerCells',
       'InnateLymphoidCells'],
      dtype='object')
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/multiVI_scFate/no4_scFate_vCM_Left.html

**Contents:**
- Read in¶
- scFates - Tree analysis¶
- Preprocessing pipeline from Palantir¶
  - Create projection dataframe¶
  - Run Palantir to obtain multiscale diffusion space¶
  - Generate embedding from the multiscale diffusion space¶
  - Plotting¶
- Tree learning with SimplePPT (or ElPiGraph)¶
  - Projecting results onto ForceAtlas2 embedding¶
  - Slecting roots and computing pseudotime¶
- Representing the trajectory and tree (on top of existing embedding)¶
- Recover full genes¶
- Save¶
  - make age continuous¶
  - save¶
- Save object separately, Compact, Trab, Gene, and ATAC¶
- Compact¶
- Trabeculated¶
- Other plots¶

ref: https://scfates.readthedocs.io/en/latest/Tree_Analysis_Bone_marrow_fates.html

16451 rows × 20 columns

**Examples:**

Example 1 (python):
```python
import scanpy as sc
import pandas as pd
import numpy as np
import anndata
import scipy

import matplotlib.pyplot as plt
import seaborn as sns
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
anndata             0.9.2
matplotlib          3.5.2
numpy               1.22.3
pandas              1.5.3
scanpy              1.8.2
scipy               1.8.1
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                 10.0.0
asttokens           NA
backcall            0.2.0
beta_ufunc          NA
binom_ufunc         NA
cffi                1.15.1
colorama            0.4.6
comm                0.1.4
cpuinfo             NA
cycler              0.10.0
cython_runtime      NA
dateutil            2.8.2
debugpy             1.6.7.post1
decorator           5.1.1
executing           1.2.0
h5py                3.9.0
hypergeom_ufunc     NA
igraph              0.10.6
importlib_metadata  NA
ipykernel           6.25.1
jedi                0.19.0
joblib              1.3.2
kiwisolver          1.4.5
leidenalg           0.10.1
llvmlite            0.40.1
mpl_toolkits        NA
natsort             8.4.0
nbinom_ufunc        NA
numba               0.57.1
numexpr             2.8.5
packaging           23.1
parso               0.8.3
patsy               0.5.3
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
platformdirs        3.10.0
plotly              5.16.1
prompt_toolkit      3.0.39
psutil              5.9.5
ptyprocess          0.7.0
pure_eval           0.2.2
pydev_ipython       NA
pydevconsole        NA
pydevd              2.9.5
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.16.1
pyparsing           3.0.9
pytz                2023.3
sinfo               0.3.4
six                 1.16.0
sklearn             1.3.0
sphinxcontrib       NA
stack_data          0.6.2
statsmodels         0.14.0
tables              3.8.0
texttable           1.6.7
threadpoolctl       3.2.0
tornado             6.3.3
traitlets           5.9.0
typing_extensions   NA
wcwidth             0.2.6
zipp                NA
zmq                 25.1.1
```

Example 5 (unknown):
```unknown
-----
IPython             8.12.2
jupyter_client      8.3.0
jupyter_core        5.3.1
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-4.15.0-197-generic-x86_64-with-glibc2.17
-----
Session information updated at 2023-09-04 09:13
```

Example 6 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 7 (python):
```python
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/vCM/MultiVI-post_foetal_vCM_onlyMultiome_allregion_onlyLeft_feature-select-001.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 16451 × 174604
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N', 'n_cells'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'leiden', 'leiden_MultiVI_colors', 'neighbors', 'region_colors', 'umap', 'week_colors'
    obsm: 'MultiVI_latent', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
sc.pl.umap(adata, color=['fine_grain','week'], size=10, wspace=0.3, components='all', ncols=3)
```

Example 10 (python):
```python
sc.pl.umap(adata, color=['week'], groups=['4W','5W','7W','8W','9W',], size=10, wspace=0.3, components='all', ncols=3)
```

Example 11 (python):
```python
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: In a future version of pandas all arguments of Categorical.replace except for the argument 'value' will be keyword-only.
  values = values.replace(values.categories.difference(groups), np.nan)
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: Categorical.replace is deprecated and will be removed in a future version. Use Series.replace directly instead.
  values = values.replace(values.categories.difference(groups), np.nan)
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: In a future version of pandas all arguments of Categorical.replace except for the argument 'value' will be keyword-only.
  values = values.replace(values.categories.difference(groups), np.nan)
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: Categorical.replace is deprecated and will be removed in a future version. Use Series.replace directly instead.
  values = values.replace(values.categories.difference(groups), np.nan)
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: In a future version of pandas all arguments of Categorical.replace except for the argument 'value' will be keyword-only.
  values = values.replace(values.categories.difference(groups), np.nan)
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: Categorical.replace is deprecated and will be removed in a future version. Use Series.replace directly instead.
  values = values.replace(values.categories.difference(groups), np.nan)
```

Example 12 (python):
```python
sc.pl.umap(adata, color=['week'], groups=['12W','13W','14W','15W','20W',], size=10, wspace=0.3, components='all', ncols=3)
```

Example 13 (python):
```python
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: In a future version of pandas all arguments of Categorical.replace except for the argument 'value' will be keyword-only.
  values = values.replace(values.categories.difference(groups), np.nan)
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: Categorical.replace is deprecated and will be removed in a future version. Use Series.replace directly instead.
  values = values.replace(values.categories.difference(groups), np.nan)
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: In a future version of pandas all arguments of Categorical.replace except for the argument 'value' will be keyword-only.
  values = values.replace(values.categories.difference(groups), np.nan)
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: Categorical.replace is deprecated and will be removed in a future version. Use Series.replace directly instead.
  values = values.replace(values.categories.difference(groups), np.nan)
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: In a future version of pandas all arguments of Categorical.replace except for the argument 'value' will be keyword-only.
  values = values.replace(values.categories.difference(groups), np.nan)
/home/jovyan/my-conda-envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:1227: FutureWarning: Categorical.replace is deprecated and will be removed in a future version. Use Series.replace directly instead.
  values = values.replace(values.categories.difference(groups), np.nan)
```

Example 14 (python):
```python
import scFates as scf
import palantir
```

Example 15 (unknown):
```unknown
findfont: Font family ['Raleway'] not found. Falling back to DejaVu Sans.
findfont: Font family ['Lato'] not found. Falling back to DejaVu Sans.
```

Example 16 (python):
```python
projections = pd.DataFrame(adata.obsm["MultiVI_latent"],index=adata.obs_names)
projections
```

Example 17 (unknown):
```unknown
# fate probability mapping of the cells
dm_res = palantir.utils.run_diffusion_maps(projections,knn=30)
ms_data = palantir.utils.determine_multiscale_space(dm_res,n_eigs=5)
```

Example 18 (unknown):
```unknown
Determing nearest neighbor graph...
```

Example 19 (python):
```python
# Generate embedding from the multiscale diffusion space
adata.obsm["X_palantir"]=ms_data.values
sc.pp.neighbors(adata,n_neighbors=30,use_rep="X_palantir")
```

Example 20 (python):
```python
sc.pl.embedding(adata, basis='X_palantir',color=['week','fine_grain'], components='all',ncols=3,wspace=0.6,size=10)
```

Example 21 (python):
```python
%%time
# draw ForceAtlas2 embedding using UMAP dim1 and dim2 as initial position
adata.obsm['X_umap13'] = adata.obsm['X_umap'][:,[0,2]]
# adata.obsm['X_palantir12'] = adata.obsm['X_palantir'][:,[0,1]]
sc.tl.draw_graph(adata,init_pos='X_umap13',random_state=0)
```

Example 22 (unknown):
```unknown
CPU times: user 2min 23s, sys: 23.1 ms, total: 2min 23s
Wall time: 2min 23s
```

Example 23 (python):
```python
# adata.uns["week_colors"][list(adata.obs['week'].cat.categories).index('13W')]='purple'
sc.pl.draw_graph(adata,color="week",color_map="RdBu_r")
```

Example 24 (python):
```python
sc.pl.draw_graph(adata,color="fine_grain",color_map="RdBu_r")
```

Example 25 (python):
```python
scf.tl.tree(adata,method="ppt",Nodes=40,use_rep="palantir",
            device="cpu",seed=5,ppt_lambda=100,ppt_sigma=0.025,ppt_nsteps=200)
```

Example 26 (unknown):
```unknown
inferring a principal tree --> parameters used 
    40 principal points, sigma = 0.025, lambda = 100, metric = euclidean
    fitting:  29%|██▉       | 58/200 [00:04<00:10, 13.10it/s]
    converged
    finished (0:00:04) --> added 
    .uns['ppt'], dictionnary containing inferred tree.
    .obsm['X_R'] soft assignment of cells to principal points.
    .uns['graph']['B'] adjacency matrix of the principal points.
    .uns['graph']['F'] coordinates of principal points in representation space.
```

Example 27 (python):
```python
scf.pl.graph(adata,tips=True,forks=False)
```

Example 28 (python):
```python
# Selecting a root and computing pseudotime
scf.tl.root(adata,17)
# generate 100 mappings of pseudotime, to account for cell asssignment uncertainty.
# .obs will be saved the mean of all calculated pseudotimes.
scf.tl.pseudotime(adata,n_jobs=1,n_map=1,seed=40)
```

Example 29 (python):
```python
node 17 selected as a root --> added
    .uns['graph']['root'] selected root.
    .uns['graph']['pp_info'] for each PP, its distance vs root and segment assignment.
    .uns['graph']['pp_seg'] segments network information.
projecting cells onto the principal graph
    finished (0:00:16) --> added
    .obs['edge'] assigned edge.
    .obs['t'] pseudotime value.
    .obs['seg'] segment of the tree assigned.
    .obs['milestones'] milestone assigned.
    .uns['pseudotime_list'] list of cell projection from all mappings.
```

Example 30 (python):
```python
scf.pl.trajectory(adata,arrows=True,arrow_offset=3,scale_path=0.5,
                 )
```

Example 31 (python):
```python
sc.pl.draw_graph(adata, color=["t","seg","milestones"], cmap='viridis')
```

Example 32 (python):
```python
sc.pl.draw_graph(adata[adata.obs['fine_grain']=='VentricularCardiomyocytesLeftTrabeculated'], 
                 color=["t"], cmap='viridis')
```

Example 33 (python):
```python
adata.obs['fine_grain'].value_counts()
```

Example 34 (unknown):
```unknown
VentricularCardiomyocytesLeftCompact         11397
VentricularCardiomyocytesLeftTrabeculated     5054
Name: fine_grain, dtype: int64
```

Example 35 (python):
```python
# read in anndata which has all the features
adata_all = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/vCM/MultiVI-pre_paired_foetal_vCMs_raw.h5ad')

# create new anndata
adata_new = anndata.AnnData(X=adata_all[adata.obs_names,:].X, # from all feature data
                      obs=adata.obs, 
                      var=adata_all.var, # from all feature data
                      uns=adata.uns,
                      obsm=adata.obsm, 
                      obsp=adata.obsp 
                               )
adata_new
```

Example 36 (unknown):
```unknown
AnnData object with n_obs × n_vars = 16451 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'umap', 'week_colors', 'seg_colors'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
```

Example 37 (python):
```python
# make time continuous
import re
def age_to_days(age_str):
    match = re.match(r'(\d+)W(\d+)D', age_str)
    if match:
        weeks, days = map(int, match.groups())
        return float(weeks * 7 + days)
    else:
        return None

# Assuming 'adata' is an AnnData object
adata_new.obs['age_days'] = adata_new.obs['age'].apply(age_to_days)
adata_new.obs['age_days'] = adata_new.obs['age_days'].astype(float)

# week
adata_new.obs['week_cont'] = [int(x.replace('W','')) for x in adata_new.obs['week']]
```

Example 38 (python):
```python
adata_new.write('/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.h5ad')
```

Example 39 (python):
```python
# Log-normalise
# gene expression
adata_lognorm = adata_new[:,adata_new.var['modality']=='Gene Expression'].copy()
sc.pp.normalize_total(adata_lognorm, target_sum=1e4)
sc.pp.log1p(adata_lognorm)
```

Example 40 (python):
```python
sc.pl.draw_graph(adata_lognorm,color=["t",'week','fine_grain'],legend_fontsize=10, cmap='viridis')
```

Example 41 (python):
```python
sc.pl.draw_graph(adata_lognorm,color=['age_days'],legend_fontsize=10, cmap='viridis')
```

Example 42 (python):
```python
sc.pl.draw_graph(adata_lognorm, color=["HEY2","MYOM2","MYH7","MYL2",'TNNI3','CPT1B','ACTN1','TBX20','CACNA1C','RYR2',
                                       'CD36','PRDM16','SCN5A','GJA1'],legend_loc='on data',legend_fontsize=10, cmap='RdPu')
```

Example 43 (python):
```python
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.h5ad')
print(adata.X.data[:10])
adata
```

Example 44 (unknown):
```unknown
[1. 2. 3. 2. 5. 3. 2. 1. 1. 1.]
```

Example 45 (unknown):
```unknown
AnnData object with n_obs × n_vars = 16451 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'seg_colors', 'umap', 'week_colors'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
```

Example 46 (python):
```python
adata.var['modality'].value_counts()
```

Example 47 (unknown):
```unknown
Peaks              508040
Gene Expression     36601
Name: modality, dtype: int64
```

Example 48 (python):
```python
adata.obs['fine_grain'].value_counts()
```

Example 49 (unknown):
```unknown
VentricularCardiomyocytesLeftCompact         11397
VentricularCardiomyocytesLeftTrabeculated     5054
Name: fine_grain, dtype: int64
```

Example 50 (python):
```python
# subset
adata_sub = adata[adata.obs_names[adata.obs['fine_grain']=='VentricularCardiomyocytesLeftCompact']]
print(adata_sub.shape)

# bin t
n_bin_time = 3
adata_sub.obs['t_binned'] = pd.cut(adata_sub.obs['t'],
                               bins=n_bin_time,
                               labels=[f't{str(x+1)}' for x in range(n_bin_time)])
print(adata_sub.obs['t_binned'].value_counts())
sc.pl.draw_graph(adata_sub,color=['t_binned'],color_map="viridis", wspace=0.2)
```

Example 51 (unknown):
```unknown
(11397, 544641)
```

Example 52 (python):
```python
/tmp/ipykernel_468/2720878155.py:7: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
  adata_sub.obs['t_binned'] = pd.cut(adata_sub.obs['t'],
```

Example 53 (unknown):
```unknown
t3    7754
t2    3477
t1     166
Name: t_binned, dtype: int64
```

Example 54 (python):
```python
# save
# RNA
adata_gex = adata_sub[:,adata_sub.var['modality']=='Gene Expression']
adata_gex.write('/nfs/team205/heart/anndata_objects/Vitro/post_multiVI_full-features/Foetal_vCM_Left-compact_rna.h5ad')
print(adata_gex)

# ATAC
adata_atac = adata_sub[:,adata_sub.var['modality']=='Peaks']
adata_atac.write('/nfs/team205/heart/anndata_objects/Vitro/post_multiVI_full-features/Foetal_vCM_Left-compact_atac.h5ad')
print(adata_atac)
```

Example 55 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 11397 × 36601
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont', 't_binned'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'umap', 'week_colors', 't_binned_colors'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
View of AnnData object with n_obs × n_vars = 11397 × 508040
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont', 't_binned'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'umap', 'week_colors', 't_binned_colors'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
```

Example 56 (python):
```python
# subset
adata_sub = adata[adata.obs_names[adata.obs['fine_grain']=='VentricularCardiomyocytesLeftTrabeculated']]
print(adata_sub.shape)

# bin t
n_bin_time = 3
adata_sub.obs['t_binned'] = pd.cut(adata_sub.obs['t'],
                               bins=n_bin_time,
                               labels=[f't{str(x+1)}' for x in range(n_bin_time)])
print(adata_sub.obs['t_binned'].value_counts())
sc.pl.draw_graph(adata_sub,color=['t_binned'],color_map="viridis", wspace=0.2)
```

Example 57 (unknown):
```unknown
(5054, 544641)
```

Example 58 (python):
```python
/tmp/ipykernel_468/2639444817.py:7: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
  adata_sub.obs['t_binned'] = pd.cut(adata_sub.obs['t'],
```

Example 59 (unknown):
```unknown
t3    3021
t2    1886
t1     147
Name: t_binned, dtype: int64
```

Example 60 (python):
```python
# save
# RNA
adata_gex = adata_sub[:,adata_sub.var['modality']=='Gene Expression']
adata_gex.write('/nfs/team205/heart/anndata_objects/Vitro/post_multiVI_full-features/Foetal_vCM_Left-trabeculated_rna.h5ad')
print(adata_gex)

# ATAC
adata_atac = adata_sub[:,adata_sub.var['modality']=='Peaks']
adata_atac.write('/nfs/team205/heart/anndata_objects/Vitro/post_multiVI_full-features/Foetal_vCM_Left-trabeculated_atac.h5ad')
print(adata_atac)
```

Example 61 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 5054 × 36601
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont', 't_binned'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'umap', 'week_colors', 't_binned_colors'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
View of AnnData object with n_obs × n_vars = 5054 × 508040
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont', 't_binned'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'umap', 'week_colors', 't_binned_colors'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
```

Example 62 (python):
```python
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.h5ad')
```

Example 63 (unknown):
```unknown
AnnData object with n_obs × n_vars = 16451 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'seg_colors', 'umap', 'week_colors'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
```

Example 64 (python):
```python
sns.scatterplot(adata.obs['t'],adata.obs['t'])
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/CellOracle/CellOracle_no1-1_construct-GRN.html

**Contents:**
- Import libraries¶
- Load gene-expression data¶
- Make oracle object¶
- Add scenicplus base GRN¶
- KNN imputation¶
- GRN calculation¶
- Network preprocessing for perturbation simulation¶
- Save the oracle object¶
- Network preprocessing¶
- Network analysis; Network score for each gene¶
- Find parameters for n_grid and min_mass for perturbation simulation¶
- Make predictive modelos for simulation¶
- In silico TF perturbation analysis¶
- Visualisation¶
- Find parameters for n_grid and min_mass¶
- Plot vector field¶
- Compare simulation vector with development vectors¶
- Focus on a single development lineage to interpret the results in detail¶

ref: https://morris-lab.github.io/CellOracle.documentation/notebooks/04_Network_analysis/Network_analysis_with_Paul_etal_2015_data.html

CellOracle uses the same strategy as velocyto for visualizing cell transitions. This process requires KNN imputation in advance.

For the KNN imputation, we first need to calculate and select PCs.

Estimate the optimal number of nearest neighbors for KNN imputation.

59522 rows × 6 columns

from below ref: https://morris-lab.github.io/CellOracle.documentation/notebooks/05_simulation/Gata1_KO_simulation_with_Paul_etal_2015_data.html

Here, we will need to fit the ridge regression models again. This process will take less time than the GRN inference in the previous notebook, because we are using the filtered GRN models.

The next step calculates the probability of cell state transitions based on the simulation data. You can use the transition probabilities between cells to predict how cells will change after a perturbation.

This transition probability will be used later.

Caution: It is very important to find the optimal scale parameter.

If the vectors are not visible, you can try a smaller scale parameter to magnify the vector length. However, if you see large vectors in the randomized results (right panel), it means that the scale parameter is too small.

Again, we need to adjust the scale parameter. Please seek to find the optimal scale parameter that provides good visualization.

If you don’t see any vector, you can try the smaller scale parameter to magnify the vector length. However, if you see large vectors in the randomized results (right panel), it means the scale parameter is too small.

Here, we will calculate the development vector field using pseudotime gradient as follows.

Note: Other methods may be used to represent a continuous scRNA-seq trajectory flow. For example, RNA velocity analysis is a good way to estimate the direction of cell differentiation. Choose the method that best suits your data.

In the analysis below, we will use pseudotime data. Pseudotime data is included in the demo data. If you are analyzing your own scRNA-seq data, please calculate pseudotime before starting this analysis.

Next, we convert the pseudotime data into grid points. For this calculation we can chose one of two methods.

knn: K-Nearesr Neighbor regression. You will need to set number of neighbors. Please adjust n_knn for best results.This will depend on the population size and density of your scRNA-seq data.

gradient.transfer_data_into_grid(args={"method": "knn", "n_knn":50})

polynomial: Polynomial regression using x-axis and y-axis of dimensional reduction space.

In general, this method will be more robust. Please use this method if knn method does not work. n_poly is the number of degrees for the polynomial regression model. Please try to find appropriaten_poly searching for best results.

gradient.transfer_data_into_grid(args={"method": "polynomial", "n_poly":3})

Calculate the 2D vector map to represents the pseudotime gradient. After the gradient calculation, the length of the vector will be normalized automatically.

Please adjust the scale parameter to adjust vector length visualization.

The inner product represents the similarity between two vectors.

Using the inner product, we compare the 2D vector field of perturbation simulation and development flow.

The inner product will be a positive value when two vectors are pointing in the same direction.

Conversely, the inner product will be a negative value when two vectors are pointing in the opposing directions.

The length of the vectors also affects the magnitude of inner product value.

In summary, we quantitatively compare the directionality and size of vectors between perturbation simulation and natural differentiation using inner product, and we define the score as perturbation score (PS).

A negative PS means that the TF perturbation would block differentiation.

A positive PS means that the TF perturbation would promote differentiation.

Here, need to adjust the vm parameter for the PS score color visualization. Please seek to find the optimal vm parameter that provides good visualization.

If you don’t see any color in the left panel below, you can try the smaller vm parameter to magnify the scale of vm visualization. However, if you see colors in the randomized results (right panel), it means the vm parameter is too small.

**Examples:**

Example 1 (python):
```python
import os
import sys

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scanpy as sc
import seaborn as sns
```

Example 2 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 3 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 4 (python):
```python
# read in adata
# whole data
adata = sc.read_h5ad('/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA/aCMs_WholeSample_paired_with_time.h5ad')
adata
```

Example 5 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
    obsp: 'connectivities', 'distances'
```

Example 6 (python):
```python
print(adata.X.data[:5])
adata.var['modality'].value_counts()
```

Example 7 (unknown):
```unknown
[2. 1. 1. 1. 1.]
```

Example 8 (unknown):
```unknown
Peaks              508040
Gene Expression     36601
Name: modality, dtype: int64
```

Example 9 (python):
```python
# subset gene expression data
adata = adata[:,adata.var['modality']=='Gene Expression']

# keep raw cont data before log normalisation
adata.raw = adata
adata.layers["raw_count"] = adata.raw.X.copy()

# filter and log-normalise
sc.pp.filter_genes(adata, min_cells=3)
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

print(adata.X.data[:10])
adata
```

Example 10 (unknown):
```unknown
[1.559231  1.0569503 1.0569503 1.0569503 1.0569503 1.0569503 2.7738788
 1.559231  1.0569503 1.0569503]
```

Example 11 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 23953
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'n_cells'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors', 'log1p'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
    layers: 'raw_count'
    obsp: 'connectivities', 'distances'
```

Example 12 (python):
```python
# select genes which are targeted or the regulators in the regulons
# 
ereg_signatures = utils.read_pkl("/nfs/team205/heart/pyscenic/plus/eRegulon_signatures/aCM_n570_Gene_eRegulon_signatures.pkl")
genes_sel = []
for key,value in ereg_signatures.items():
    genes_sel = genes_sel + value
    genes_sel = genes_sel + [key.split('_')[0]]
genes_sel = list(set(genes_sel))

# hvg and targeted
# hvg_targeted = list(set(adata.var_names[adata.var.highly_variable]).intersection(genes_targeted))
# len(hvg_targeted)
len(genes_sel)
```

Example 13 (python):
```python
# subset genes
# adata = adata[:,hvg_targeted]
adata = adata[:,genes_sel]
adata
```

Example 14 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 5664 × 10870
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'n_cells'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors', 'log1p'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
    layers: 'raw_count'
    obsp: 'connectivities', 'distances'
```

Example 15 (python):
```python
print(f"Cell number is :{adata.shape[0]}")
print(f"Gene number is :{adata.shape[1]}")
```

Example 16 (unknown):
```unknown
Cell number is :5664
Gene number is :10870
```

Example 17 (python):
```python
# add t_binned category
n_bin_time = 3
adata.obs['t_binned'] = pd.cut(adata.obs['t'],
                               bins=n_bin_time,
                               labels=[f't{str(x+1)}' for x in range(n_bin_time)])
print(set(adata.obs['t_binned']))

# combine fine_grain and t_binned category
adata.obs['fine_grain_time-binned'] = adata.obs['fine_grain'].astype('str')+'_'+adata.obs['t_binned'].astype('str')
```

Example 18 (python):
```python
/tmp/ipykernel_3282310/3161589083.py:3: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
  adata.obs['t_binned'] = pd.cut(adata.obs['t'],
```

Example 19 (unknown):
```unknown
{'t1', 't2', 't3'}
```

Example 20 (python):
```python
sc.pl.umap(adata,color=['fine_grain','t','fine_grain_time-binned'], size=5, wspace=0.3, ncols=3)
```

Example 21 (python):
```python
import celloracle as co
co.__version__
```

Example 22 (unknown):
```unknown
2025-05-05 21:07:46,185 - INFO - BioProspector no longer found.
2025-05-05 21:07:46,186 - INFO - HMS no longer found.
2025-05-05 21:07:46,188 - INFO - MDmodule no longer found.
2025-05-05 21:07:46,189 - INFO - Posmo no longer found.
2025-05-05 21:07:46,194 - INFO - Using included version of BioProspector.
2025-05-05 21:07:46,198 - INFO - Using included version of HMS.
2025-05-05 21:07:46,202 - INFO - Using included version of MDmodule.
2025-05-05 21:07:46,205 - INFO - Using included version of Posmo.
2025-05-05 21:07:46,211 - INFO - Configuration file: /nfs/users/nfs_k/kk18/.config/gimmemotifs/gimmemotifs.cfg
```

Example 23 (unknown):
```unknown
# Instantiate Oracle object
oracle = co.Oracle()
```

Example 24 (python):
```python
# Check data in anndata
print("Metadata columns :", list(adata.obs.columns))
print("Dimensional reduction: ", list(adata.obsm.keys()))
```

Example 25 (python):
```python
Metadata columns : ['sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones', 't_binned', 'fine_grain_time-binned']
Dimensional reduction:  ['X_R', 'X_multiVI', 'X_umap']
```

Example 26 (python):
```python
# In this notebook, we use the unscaled mRNA count for the nput of Oracle object.
adata.X = adata.layers["raw_count"].copy()

# Instantiate Oracle object.
oracle.import_anndata_as_raw_count(adata=adata,
                                   cluster_column_name="fine_grain_time-binned",
                                   embedding_name="X_umap")
```

Example 27 (python):
```python
10870 genes were found in the adata. Note that Celloracle is intended to use around 1000-3000 genes, so the behavior with this number of genes may differ from what is expected.
WARNING: adata.X seems to be already log-transformed.
```

Example 28 (unknown):
```unknown
# read in scenicplus regulons
regulons = utils.read_pkl('/nfs/team205/heart/pyscenic/plus/baseGRN_forCO/AtrialCM_baseGRN_act.pkl')
TG_to_TF_dictionary = co.utility.inverse_dictionary(regulons)
```

Example 29 (unknown):
```unknown
0%|          | 0/10836 [00:00<?, ?it/s]
```

Example 30 (unknown):
```unknown
# add
oracle.import_TF_data(TFdict=TG_to_TF_dictionary)
```

Example 31 (unknown):
```unknown
TG_to_TF_dictionary['SGSM2']
```

Example 32 (unknown):
```unknown
['KLF13', 'KLF15', 'MBD1', 'RELA', 'SREBF1', 'VEZF1', 'ZNF28', 'ZNF644']
```

Example 33 (python):
```python
%%time
%matplotlib inline
# Perform PCA
oracle.perform_PCA()

# Select important PCs
plt.plot(np.cumsum(oracle.pca.explained_variance_ratio_)[:100])
n_comps = np.where(np.diff(np.diff(np.cumsum(oracle.pca.explained_variance_ratio_))>0.002))[0][0]
plt.axvline(n_comps, c="k")
plt.show()
print(n_comps)
n_comps = min(n_comps, 50)
```

Example 34 (unknown):
```unknown
10
CPU times: user 1min 32s, sys: 994 ms, total: 1min 33s
Wall time: 1min 36s
```

Example 35 (python):
```python
oracle.pcs.shape
```

Example 36 (unknown):
```unknown
(5664, 5664)
```

Example 37 (python):
```python
n_cell = oracle.adata.shape[0]
print(f"cell number is :{n_cell}")
```

Example 38 (unknown):
```unknown
cell number is :5664
```

Example 39 (python):
```python
k = int(0.025*n_cell)
print(f"Auto-selected k is :{k}")
```

Example 40 (unknown):
```unknown
Auto-selected k is :141
```

Example 41 (unknown):
```unknown
%%time
oracle.knn_imputation(n_pca_dims=n_comps, k=k, balanced=True, b_sight=k*8,
                      b_maxl=k*4, n_jobs=4)
```

Example 42 (unknown):
```unknown
CPU times: user 8.42 s, sys: 1.53 s, total: 9.94 s
Wall time: 10.2 s
```

Example 43 (unknown):
```unknown
# Save oracle object.
oracle.to_hdf5("/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM.celloracle.oracle")
```

Example 44 (unknown):
```unknown
# Load
oracle = co.load_hdf5("/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM.celloracle.oracle")
```

Example 45 (python):
```python
%matplotlib inline
# Check clustering data
sc.pl.umap(oracle.adata, color='fine_grain_time-binned')
```

Example 46 (unknown):
```unknown
%%time
# Calculate GRN for each population in "louvain_annot" clustering unit.
# This step may take some time.(~30 minutes)
links = oracle.get_links(cluster_name_for_GRN_unit='fine_grain_time-binned', alpha=10,
                         verbose_level=10)
```

Example 47 (unknown):
```unknown
0%|          | 0/6 [00:00<?, ?it/s]
```

Example 48 (unknown):
```unknown
Inferring GRN for AtrialCardiomyocytesLeft_t1...
```

Example 49 (unknown):
```unknown
0%|          | 0/10836 [00:00<?, ?it/s]
```

Example 50 (unknown):
```unknown
Inferring GRN for AtrialCardiomyocytesLeft_t2...
```

Example 51 (unknown):
```unknown
0%|          | 0/10836 [00:00<?, ?it/s]
```

Example 52 (unknown):
```unknown
Inferring GRN for AtrialCardiomyocytesLeft_t3...
```

Example 53 (unknown):
```unknown
0%|          | 0/10836 [00:00<?, ?it/s]
```

Example 54 (unknown):
```unknown
Inferring GRN for AtrialCardiomyocytesRight_t1...
```

Example 55 (unknown):
```unknown
0%|          | 0/10836 [00:00<?, ?it/s]
```

Example 56 (unknown):
```unknown
Inferring GRN for AtrialCardiomyocytesRight_t2...
```

Example 57 (unknown):
```unknown
0%|          | 0/10836 [00:00<?, ?it/s]
```

Example 58 (unknown):
```unknown
Inferring GRN for AtrialCardiomyocytesRight_t3...
```

Example 59 (unknown):
```unknown
0%|          | 0/10836 [00:00<?, ?it/s]
```

Example 60 (unknown):
```unknown
CPU times: user 54min 47s, sys: 12.5 s, total: 55min
Wall time: 56min 58s
```

Example 61 (unknown):
```unknown
links.links_dict.keys()
```

Example 62 (unknown):
```unknown
dict_keys(['AtrialCardiomyocytesLeft_t1', 'AtrialCardiomyocytesLeft_t2', 'AtrialCardiomyocytesLeft_t3', 'AtrialCardiomyocytesRight_t1', 'AtrialCardiomyocytesRight_t2', 'AtrialCardiomyocytesRight_t3'])
```

Example 63 (unknown):
```unknown
links.links_dict['AtrialCardiomyocytesRight_t3']
```

Example 64 (unknown):
```unknown
# Save Links object.
links.to_hdf5(file_path="/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM_links.celloracle.links")
```

Example 65 (unknown):
```unknown
# load the saved links
links = co.load_hdf5("/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM_links.celloracle.links")
```

Example 66 (unknown):
```unknown
%%time

links.filter_links(p=0.001, weight="coef_abs", threshold_number=10000)

# Calculate network scores.
links.get_network_score()
display(links.merged_score.head())

# Make predictive modelos for simulation
# Here, we will need to fit the ridge regression models again.
# This process will take less time than the GRN inference in the previous notebook, because we are using the filtered GRN models.
oracle.get_cluster_specific_TFdict_from_Links(links_object=links)
oracle.fit_GRN_for_simulation(alpha=10,
                              use_cluster_specific_TFdict=True)
```

Example 67 (unknown):
```unknown
0%|          | 0/6 [00:00<?, ?it/s]
```

Example 68 (unknown):
```unknown
CPU times: user 3min 12s, sys: 3.54 s, total: 3min 15s
Wall time: 3min 21s
```

Example 69 (unknown):
```unknown
oracle.to_hdf5("/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM.celloracle.oracle")
oracle
```

Example 70 (unknown):
```unknown
Oracle object

Meta data
    celloracle version used for instantiation: 0.20.0
    n_cells: 5664
    n_genes: 10870
    cluster_name: fine_grain_time-binned
    dimensional_reduction_name: X_umap
    n_target_genes_in_TFdict: 10836 genes
    n_regulatory_in_TFdict: 263 genes
    n_regulatory_in_both_TFdict_and_scRNA-seq: 263 genes
    n_target_genes_both_TFdict_and_scRNA-seq: 10836 genes
    k_for_knn_imputation: 141
Status
    Gene expression matrix: Ready
    BaseGRN: Ready
    PCA calculation: Done
    Knn imputation: Done
    GRN calculation for simulation: Done
```

Example 71 (unknown):
```unknown
links.filter_links(p=0.001, weight="coef_abs", threshold_number=10000)
```

Example 72 (python):
```python
plt.rcParams["figure.figsize"] = [7, 3]
```

Example 73 (unknown):
```unknown
links.plot_degree_distributions(plot_model=True,
                                               #save=f"{save_folder}/degree_distribution/",
                                               )
```

Example 74 (unknown):
```unknown
AtrialCardiomyocytesLeft_t1
```

Example 75 (unknown):
```unknown
AtrialCardiomyocytesLeft_t2
```

Example 76 (unknown):
```unknown
AtrialCardiomyocytesLeft_t3
```

Example 77 (unknown):
```unknown
AtrialCardiomyocytesRight_t1
```

Example 78 (unknown):
```unknown
AtrialCardiomyocytesRight_t2
```

Example 79 (unknown):
```unknown
AtrialCardiomyocytesRight_t3
```

Example 80 (unknown):
```unknown
# Calculate network scores.
links.get_network_score()
```

Example 81 (unknown):
```unknown
links.merged_score.head()
```

Example 82 (unknown):
```unknown
# Check cluster name
links.cluster
```

Example 83 (unknown):
```unknown
['AtrialCardiomyocytesLeft_t1',
 'AtrialCardiomyocytesLeft_t2',
 'AtrialCardiomyocytesLeft_t3',
 'AtrialCardiomyocytesRight_t1',
 'AtrialCardiomyocytesRight_t2',
 'AtrialCardiomyocytesRight_t3']
```

Example 84 (python):
```python
# Visualize top n-th genes with high scores.
plt.rcParams["figure.figsize"] = [5, 7]
links.plot_scores_as_rank(cluster='AtrialCardiomyocytesRight_t3', n_gene=30, save=None)
```

Example 85 (python):
```python
# Visualize top n-th genes with high scores.
plt.rcParams["figure.figsize"] = [5, 7]
links.plot_scores_as_rank(cluster='AtrialCardiomyocytesLeft_t3', n_gene=30, save=None)
```

Example 86 (python):
```python
# Compare GRN score between two clusters
plt.rcParams["figure.figsize"] = [5, 5]
links.plot_score_comparison_2D(value="betweenness_centrality",
                               cluster1='AtrialCardiomyocytesRight_t3', cluster2='AtrialCardiomyocytesRight_t1',
                               percentile=98,
                               save=None)
```

Example 87 (unknown):
```unknown
links.plot_score_comparison_2D(value="betweenness_centrality",
                               cluster1='AtrialCardiomyocytesLeft_t3', cluster2='AtrialCardiomyocytesLeft_t1',
                               percentile=98,
                               save=None)
```

Example 88 (unknown):
```unknown
links.plot_score_comparison_2D(value="betweenness_centrality",
                               cluster1='AtrialCardiomyocytesLeft_t3', cluster2='AtrialCardiomyocytesRight_t3',
                               percentile=98,
                               save=None)
```

Example 89 (unknown):
```unknown
# set a gene for this trial
goi = "PITX2"
# Enter perturbation conditions to simulate signal propagation after the perturbation.
oracle.simulate_shift(perturb_condition={goi: 0.0}, # this is KO simulation so expression=0
                      n_propagation=3)
```

Example 90 (unknown):
```unknown
%%time
# Get transition probability
oracle.estimate_transition_prob(n_neighbors=200,
                                knn_random=True,
                                sampled_fraction=1)

# Calculate embedding
oracle.calculate_embedding_shift(sigma_corr=0.05)
```

Example 91 (python):
```python
# visualisation
fig, ax = plt.subplots(1, 2,  figsize=[13, 6])

scale = 20
# Show quiver plot
oracle.plot_quiver(scale=scale, ax=ax[0])
ax[0].set_title(f"Simulated cell identity shift vector: {goi} KO")

# Show quiver plot that was calculated with randomized graph.
oracle.plot_quiver_random(scale=scale, ax=ax[1])
ax[1].set_title(f"Randomized simulation vector")

plt.show()
```

Example 92 (unknown):
```unknown
# n_grid = 40 is a good starting value.
n_grid = 40
oracle.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=200)
```

Example 93 (unknown):
```unknown
# Search for best min_mass.
oracle.suggest_mass_thresholds(n_suggestion=12)
```

Example 94 (unknown):
```unknown
min_mass = 0.011
oracle.calculate_mass_filter(min_mass=min_mass, plot=True)
```

Example 95 (unknown):
```unknown
%%time
links.filter_links()
oracle.get_cluster_specific_TFdict_from_Links(links_object=links)
oracle.fit_GRN_for_simulation(alpha=10,
                              use_cluster_specific_TFdict=True)
```

Example 96 (python):
```python
# Check gene expression
goi = "TBX20"
sc.pl.draw_graph(oracle.adata, color=[goi, oracle.cluster_column_name],
                 layer="imputed_count", use_raw=False, cmap="viridis")
```

Example 97 (python):
```python
# Plot gene expression in histogram
sc.get.obs_df(oracle.adata, keys=[goi], layer="imputed_count").hist()
plt.show()
```

Example 98 (unknown):
```unknown
# Enter perturbation conditions to simulate signal propagation after the perturbation.
oracle.simulate_shift(perturb_condition={goi: 0.0}, # this is KO simulation so expression=0
                      n_propagation=3)
```

Example 99 (unknown):
```unknown
%%time
# Get transition probability
oracle.estimate_transition_prob(n_neighbors=200,
                                knn_random=True,
                                sampled_fraction=1)

# Calculate embedding
oracle.calculate_embedding_shift(sigma_corr=0.05)
```

Example 100 (python):
```python
fig, ax = plt.subplots(1, 2,  figsize=[13, 6])

scale = 20
# Show quiver plot
oracle.plot_quiver(scale=scale, ax=ax[0])
ax[0].set_title(f"Simulated cell identity shift vector: {goi} KO")

# Show quiver plot that was calculated with randomized graph.
oracle.plot_quiver_random(scale=scale, ax=ax[1])
ax[1].set_title(f"Randomized simulation vector")

plt.show()
```

Example 101 (unknown):
```unknown
# n_grid = 40 is a good starting value.
n_grid = 40
oracle.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=200)
```

Example 102 (unknown):
```unknown
# Search for best min_mass.
oracle.suggest_mass_thresholds(n_suggestion=12)
```

Example 103 (unknown):
```unknown
min_mass = 0.011
oracle.calculate_mass_filter(min_mass=min_mass, plot=True)
```

Example 104 (python):
```python
fig, ax = plt.subplots(1, 2,  figsize=[13, 6])

scale_simulation = 0.1
# Show quiver plot
oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax[0])
ax[0].set_title(f"Simulated cell identity shift vector: {goi} KO")

# Show quiver plot that was calculated with randomized graph.
oracle.plot_simulation_flow_random_on_grid(scale=scale_simulation, ax=ax[1])
ax[1].set_title(f"Randomized simulation vector")

plt.show()
```

Example 105 (python):
```python
# Plot vector field with cell cluster
fig, ax = plt.subplots(figsize=[8, 8])

oracle.plot_cluster_whole(ax=ax, s=10)
oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax, show_background=False)
```

Example 106 (python):
```python
# Visualize pseudotime
fig, ax = plt.subplots(figsize=[6,6])

sc.pl.embedding(adata=oracle.adata, basis=oracle.embedding_name, ax=ax, cmap="rainbow",
                color=["t"])
```

Example 107 (python):
```python
from celloracle.applications import Gradient_calculator

# Instantiate Gradient calculator object
gradient = Gradient_calculator(oracle_object=oracle, pseudotime_key="t")
```

Example 108 (unknown):
```unknown
gradient.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=200)
gradient.calculate_mass_filter(min_mass=min_mass, plot=True)
```

Example 109 (unknown):
```unknown
gradient.transfer_data_into_grid(args={"method": "knn", "n_knn":50})
```

Example 110 (unknown):
```unknown
# Calculate graddient
gradient.calculate_gradient()

# Show results
scale_dev = 40
gradient.visualize_results(scale=scale_dev, s=5)
```

Example 111 (python):
```python
# Visualize results
fig, ax = plt.subplots(figsize=[6, 6])
gradient.plot_dev_flow_on_grid(scale=scale_dev, ax=ax)
```

Example 112 (python):
```python
from celloracle.applications import Oracle_development_module

# Make Oracle_development_module to compare two vector field
dev = Oracle_development_module()

# Load development flow
dev.load_differentiation_reference_data(gradient_object=gradient)

# Load simulation result
dev.load_perturb_simulation_data(oracle_object=oracle)


# Calculate inner produc scores
dev.calculate_inner_product()
dev.calculate_digitized_ip(n_bins=10)
```

Example 113 (python):
```python
# Show perturbation scores
vm = 0.01

fig, ax = plt.subplots(1, 2, figsize=[12, 6])
dev.plot_inner_product_on_grid(vm=vm, s=50, ax=ax[0])
ax[0].set_title(f"PS")

dev.plot_inner_product_random_on_grid(vm=vm, s=50, ax=ax[1])
ax[1].set_title(f"PS calculated with Randomized simulation vector")
plt.show()
```

Example 114 (python):
```python
# Show perturbation scores with perturbation simulation vector field
fig, ax = plt.subplots(figsize=[6, 6])
dev.plot_inner_product_on_grid(vm=vm, s=50, ax=ax)
dev.plot_simulation_flow_on_grid(scale=scale_simulation, show_background=False, ax=ax)
```

Example 115 (unknown):
```unknown
# Let's visualize the results
dev.visualize_development_module_layout_0(s=5,
                                          scale_for_simulation=scale_simulation,
                                          s_grid=50,
                                          scale_for_pseudotime=scale_dev,
                                          vm=vm)
```

Example 116 (unknown):
```unknown
dev.inner_product_df
```

Example 117 (unknown):
```unknown
dev.cell_idx_use
```

Example 118 (python):
```python
# Get cell index list for the cells of interest
clusters = ['VentricularCardiomyocytesLeftCompact_t1',
 'VentricularCardiomyocytesLeftCompact_t2',
 'VentricularCardiomyocytesLeftCompact_t3']
cell_idx = np.where(oracle.adata.obs["fine_grain_time-binned"].isin(clusters))[0]

# Check index
print(cell_idx)
```

Example 119 (unknown):
```unknown
dev = Oracle_development_module()

# Load development flow
dev.load_differentiation_reference_data(gradient_object=gradient)

# Load simulation result
dev.load_perturb_simulation_data(oracle_object=oracle,
                                 cell_idx_use=cell_idx, # Enter cell id list
                                 name="Lineage_Compact" # Name of this cell group. You can enter any name.
                                 )

# Calculation
dev.calculate_inner_product()
dev.calculate_digitized_ip(n_bins=10)
```

Example 120 (unknown):
```unknown
# Let's visualize the results
dev.visualize_development_module_layout_0(s=5,
                                          scale_for_simulation=scale_simulation,
                                          s_grid=50,
                                          scale_for_pseudotime=scale_dev,
                                          vm=vm)
```

Example 121 (unknown):
```unknown
dev.inner_product_df
```

Example 122 (unknown):
```unknown
df = dev.inner_product_df.copy()
df[['score','pseudotime']].corr(method='pearson')
```

Example 123 (unknown):
```unknown
dev.inner_product_df
```

Example 124 (unknown):
```unknown
len(dev.inner_product)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/multiVI_scFate/no3_post-MultiVI_vCM_Left.html

**Contents:**
- Post-multiVI, Find neighbors, UMAP/Clustering¶
- Assess which MultiVI latent space most correlate with age¶
- Save¶
- Read in (for plotting)¶

**Examples:**

Example 1 (python):
```python
import scanpy as sc
import pandas as pd
import numpy as np
import anndata
import scipy

from scipy.io import mmwrite, mmread
from scipy.sparse import csr_matrix

import sctk
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (unknown):
```unknown
-----
anndata             0.8.0
numpy               1.21.5
pandas              1.3.5
scanpy              1.8.2
scipy               1.7.3
sctk                NA
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                 9.1.1
asciitree           NA
backcall            0.2.0
beta_ufunc          NA
binom_ufunc         NA
cached_property     1.5.1
cairo               1.20.0
cffi                1.14.4
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.2
dask                2022.01.0
dateutil            2.8.1
decorator           4.4.2
defusedxml          0.6.0
fasteners           0.17.3
fsspec              2022.01.0
google              NA
h5py                3.1.0
igraph              0.10.2
importlib_metadata  1.7.0
ipykernel           5.4.2
ipython_genutils    0.2.0
ipywidgets          7.6.3
jedi                0.17.2
jinja2              3.1.2
joblib              1.0.0
kiwisolver          1.3.1
leidenalg           0.9.0
llvmlite            0.35.0
louvain             0.7.0
markupsafe          2.1.1
matplotlib          3.5.2
mpl_toolkits        NA
msgpack             1.0.3
natsort             7.1.0
nbinom_ufunc        NA
numba               0.52.0
numcodecs           0.9.1
numexpr             2.7.2
numpy_groupies      0.9.13
packaging           21.3
parso               0.7.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
plotly              5.10.0
prompt_toolkit      3.0.10
psutil              5.9.0
ptyprocess          0.7.0
pyarrow             9.0.0
pygments            2.13.0
pyparsing           2.4.7
pytz                2020.5
scvelo              0.2.4
seaborn             0.11.1
setuptools_scm      NA
simplejson          3.19.1
sinfo               0.3.1
six                 1.14.0
sklearn             1.0.2
sphinxcontrib       NA
statsmodels         0.12.1
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
threadpoolctl       2.1.0
tlz                 0.11.2
toolz               0.11.2
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
wcwidth             0.2.5
yaml                5.4.1
zarr                2.11.3
zipp                NA
zmq                 21.0.1
zope                NA
```

Example 5 (unknown):
```unknown
-----
IPython             7.19.0
jupyter_client      6.1.11
jupyter_core        4.7.0
notebook            6.2.0
-----
Python 3.7.9 | packaged by conda-forge | (default, Dec  9 2020, 21:08:20) [GCC 9.3.0]
Linux-4.15.0-112-generic-x86_64-with-debian-bullseye-sid
-----
Session information updated at 2023-08-16 07:45
```

Example 6 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 7 (unknown):
```unknown
!ls /nfs/team205/heart/anndata_objects/Foetal/vCM
```

Example 8 (unknown):
```unknown
MultiVI-model_foetal_vCM_feature-select-001
MultiVI-model_foetal_vCM_onlyMultiome_allregion_feature-select-001
MultiVI-model_foetal_vCM_onlyMultiome_allregion_onlyLeft_feature-select-001
MultiVI-model_foetal_vCM_onlyMultiome_allregion_withMouse_feature-select-001
MultiVI-model_foetal_vCM_onlyMultiome_allregion_woCycling_feature-select-001
MultiVI-model_foetal_vCM_onlyMultiome_allregion_woCycling_withMouse_feature-select-001
MultiVI-model_foetal_vCM_onlyMultiome_feature-select-001
MultiVI-model_foetal_vCM_onlyMultiome_only-whole_feature-select-001
MultiVI-post_foetal_vCM_feature-select-001.h5ad
MultiVI-post_foetal_vCM_onlyMultiome_allregion_feature-select-001.h5ad
MultiVI-post_foetal_vCM_onlyMultiome_allregion_onlyLeft_feature-select-001.h5ad
MultiVI-post_foetal_vCM_onlyMultiome_allregion_withMouse_feature-select-001.h5ad
MultiVI-post_foetal_vCM_onlyMultiome_allregion_woCycling_feature-select-001.h5ad
MultiVI-post_foetal_vCM_onlyMultiome_allregion_woCycling_withMouse_feature-select-001.h5ad
MultiVI-post_foetal_vCM_onlyMultiome_feature-select-001.h5ad
MultiVI-post_foetal_vCM_onlyMultiome_only-whole_feature-select-001.h5ad
MultiVI-pre_mouse-organogenesis_RNA_raw.h5ad
MultiVI-pre_paired_foetal_vCMs_raw.h5ad
MultiVI-pre_scRNA_foetal_vCMs_raw.h5ad
```

Example 9 (python):
```python
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/vCM/MultiVI-post_foetal_vCM_onlyMultiome_allregion_onlyLeft_feature-select-001.h5ad')
adata
```

Example 10 (unknown):
```unknown
AnnData object with n_obs × n_vars = 16451 × 174604
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N', 'n_cells'
    uns: '_scvi_manager_uuid', '_scvi_uuid'
    obsm: 'MultiVI_latent', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
```

Example 11 (python):
```python
adata.X.data[:10]
```

Example 12 (unknown):
```unknown
array([1., 1., 1., 1., 3., 1., 2., 1., 1., 1.], dtype=float32)
```

Example 13 (python):
```python
'''
# log-normalise
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
'''
```

Example 14 (python):
```python
'\n# log-normalise\nsc.pp.normalize_total(adata, target_sum=1e4)\nsc.pp.log1p(adata)\n'
```

Example 15 (python):
```python
%%time
sc.pp.neighbors(adata, use_rep="MultiVI_latent",n_neighbors=15)
sc.tl.umap(adata, min_dist=0.2, spread=1.0, n_components=3)
sc.tl.leiden(adata, resolution=0.5, key_added='leiden_MultiVI')
```

Example 16 (unknown):
```unknown
CPU times: user 42.1 s, sys: 7.83 s, total: 50 s
Wall time: 41.6 s
```

Example 17 (python):
```python
sc.pl.umap(adata, color=['leiden_MultiVI'], size=10, wspace=0.3, components='all', ncols=3)
```

Example 18 (python):
```python
sc.pl.umap(adata, color=['donor'], size=10, wspace=0.5, components='all', ncols=3)
```

Example 19 (python):
```python
sc.pl.umap(adata, color=['total_counts','pct_counts_mt','pct_counts_ribo',
                                                          'S_score','G2M_score'], 
           size=10, wspace=0.5, components='all', ncols=3, cmap='RdPu')
```

Example 20 (python):
```python
sc.pl.umap(adata, color=['fine_grain'], size=10, wspace=0.5, components='all', ncols=3)
```

Example 21 (python):
```python
for fine in [
    # 'VentricularCardiomyocytesCycling',
    # 'VentricularCardiomyocytesRightTrabeculated', 
    # 'VentricularCardiomyocytesRightCompact', 
 'VentricularCardiomyocytesLeftTrabeculated', 
 'VentricularCardiomyocytesLeftCompact', 
 ]:
    sc.pl.umap(adata, color=['fine_grain'], groups=[fine], size=10, wspace=0.5, components='all', ncols=3)
```

Example 22 (python):
```python
sc.pl.umap(adata, color=['cycling'], size=10, wspace=0.5, components='all', ncols=3)
```

Example 23 (python):
```python
sc.pl.umap(adata, color=['week'], size=10, wspace=0.5, components='all', ncols=3)
```

Example 24 (python):
```python
sc.pl.umap(adata, color=['region'], size=10, wspace=0.5, components='all', ncols=3)
```

Example 25 (python):
```python
adata.obs['week']
```

Example 26 (unknown):
```unknown
barcode
BHF_F_Hea11933669_BHF_F_Hea11596622_AATTAGCGTTAGCATG-1      13W
HCAHeartST13386009_HCAHeartST13303419_ATCCTTAGTGGATTGC-1    20W
BHF_F_Hea13188324_BHF_F_Hea13187626_CGAGGCAAGGCTGTCA-1      20W
BHF_F_Hea13188324_BHF_F_Hea13187626_TAGTAAGCACATACTG-1      20W
BHF_F_Hea13188324_BHF_F_Hea13187626_GTTTAACCACGTAATT-1      20W
                                                           ... 
BHF_F_Hea11933669_BHF_F_Hea11596622_GCCCAAATCACATTGA-1      13W
BHF_F_Hea11933667_BHF_F_Hea11596620_CGCAATTAGGTCTTGG-1      13W
BHF_F_Hea11933667_BHF_F_Hea11596620_GCGCTTGCAGGATGGC-1      13W
BHF_F_Hea11933667_BHF_F_Hea11596620_CTCTTGATCATGCGTG-1      13W
BHF_F_Hea11933668_BHF_F_Hea11596621_AACTAGCTCTTGAGGT-1      13W
Name: week, Length: 16451, dtype: category
Categories (10, object): ['4W', '5W', '7W', '8W', ..., '13W', '14W', '15W', '20W']
```

Example 27 (python):
```python
# get latent space
latent = pd.DataFrame(adata.obsm['MultiVI_latent'])
latent.index = adata.obs_names

# get age (week)
latent['week'] = [int(x.replace('W','')) for x in adata.obs['week']]
# arr = pd.Series([int(x.replace('W','')) for x in adata.obs['week']])

# correlation with week
# res = latent.corrwith(arr, axis=1, method='pearson')
res = latent.corr(method='pearson')['week']
res.sort_values(ascending=False)
```

Example 28 (unknown):
```unknown
week    1.000000
9       0.337589
2       0.233910
1       0.151557
0       0.146214
18      0.123042
10      0.106890
8       0.075663
13     -0.007618
19     -0.033006
6      -0.037545
11     -0.048397
7      -0.091645
15     -0.115572
16     -0.128860
14     -0.142237
12     -0.204365
3      -0.205714
5      -0.224725
4      -0.259518
17     -0.261819
Name: week, dtype: float64
```

Example 29 (unknown):
```unknown
latent.index = [f'cell{x}' for x in range(len(latent))]
latent.to_csv('test.csv')
```

Example 30 (python):
```python
adata.write('/nfs/team205/heart/anndata_objects/Foetal/vCM/MultiVI-post_foetal_vCM_onlyMultiome_allregion_onlyLeft_feature-select-001.h5ad')
```

Example 31 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 32 (python):
```python
# adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Vitro/multiVI/MultiVI_Foetal_Adult_CMs_Multiome_feature-select-001.h5ad')
# adata
```

Example 33 (python):
```python
sc.pl.umap(adata, color=['source'], size=5, wspace=0.5, ncols=1, frameon=False)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/Visium_mapping/c2l_pt1_reference_FFPE.html

**Contents:**
- Loading packages¶
- Functions¶
- Loading scRNA-seq reference data¶
- Make sure .var_names are ENSEMBL IDs¶
- Reduce to only those features present in the FFPE probeset¶
- Inspect batch key¶
- Make reference signatures¶

297473 rows × 1 columns

**Examples:**

Example 1 (unknown):
```unknown
run = "20Mar2024_fetal_FFPE"
```

Example 2 (unknown):
```unknown
celltype_key = 'fine_grain_modified'
```

Example 3 (python):
```python
# create paths and names to results folders for reference regression and cell2location models
import os
results_folder = os.path.join(os.getcwd(),"analysis/Visium/cell2location_results")
ref_run_name = f'{results_folder}/reference_signatures/{run}'
run_name = f'{results_folder}/cell2location_map/{run}'

os.makedirs(ref_run_name, exist_ok=True)
os.makedirs(run_name, exist_ok=True)
```

Example 4 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 5 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Global seed set to 0
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 6 (python):
```python
import warnings
# Suppress UserWarnings
warnings.filterwarnings('ignore', category=UserWarning)
```

Example 7 (python):
```python
import session_info
session_info.show()
```

Example 8 (python):
```python
-----
anndata             0.8.0
cell2location       0.1.3
matplotlib          3.7.2
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
scvi                0.20.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 9 (unknown):
```unknown
PIL                         10.0.0
absl                        NA
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.07.22
cffi                        1.15.1
charset_normalizer          3.2.0
chex                        0.1.7
comm                        0.1.3
contextlib2                 NA
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.7
decorator                   5.1.1
defusedxml                  0.7.1
docrep                      0.3.2
etils                       1.4.0
executing                   1.2.0
fastjsonschema              NA
flax                        0.7.0
fqdn                        NA
fsspec                      2023.6.0
graphviz                    0.20.1
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_resources         NA
ipykernel                   6.25.0
isoduration                 NA
jax                         0.4.14
jaxlib                      0.4.14
jedi                        0.18.2
jinja2                      3.1.2
joblib                      1.3.1
json5                       NA
jsonpointer                 2.0
jsonschema                  4.18.4
jsonschema_specifications   NA
jupyter_events              0.6.3
jupyter_server              2.7.0
jupyterlab_server           2.24.0
kiwisolver                  1.4.4
leidenalg                   0.10.1
lightning_fabric            1.9.5
lightning_utilities         0.9.0
llvmlite                    0.40.1
markupsafe                  2.1.3
ml_collections              NA
ml_dtypes                   0.2.0
mpl_toolkits                NA
mpmath                      1.3.0
msgpack                     1.0.5
mudata                      0.2.3
multipledispatch            0.6.0
natsort                     8.4.0
nbformat                    5.9.1
numba                       0.57.1
numpyro                     0.12.1
nvfuser                     NA
opt_einsum                  v3.3.0
optax                       0.1.7
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
platformdirs                3.9.1
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pyarrow                     12.0.1
pycparser                   2.21
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.15.1
pynndescent                 0.5.10
pyparsing                   3.0.9
pyro                        1.8.6
pythonjsonlogger            NA
pytorch_lightning           1.9.5
pytz                        2023.3
referencing                 NA
regex                       2.5.129
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rich                        NA
rpds                        NA
scipy                       1.11.1
send2trash                  NA
setuptools                  68.0.0
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
sparse                      0.14.0
stack_data                  0.6.2
statsmodels                 0.14.0
sympy                       1.12
texttable                   1.6.7
threadpoolctl               3.2.0
toolz                       0.12.0
torch                       2.0.1+cu117
torchmetrics                1.0.1
tornado                     6.3.2
tqdm                        4.65.0
traitlets                   5.9.0
tree                        0.1.8
typing_extensions           NA
umap                        0.5.3
uri_template                NA
urllib3                     1.26.16
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.1
yaml                        6.0
zmq                         25.1.0
```

Example 10 (unknown):
```unknown
-----
IPython             8.14.0
jupyter_client      8.3.0
jupyter_core        5.3.1
jupyterlab          4.0.3
notebook            7.0.0
-----
Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0]
Linux-5.4.0-167-generic-x86_64-with-glibc2.27
-----
Session information updated at 2023-12-15 15:04
```

Example 11 (python):
```python
# Importing necessary libraries
from cell2location.utils.filtering import filter_genes
from cell2location.models import RegressionModel

# Function to make c2l refernce
def c2l_make_reference(adata, subset_value,max_epochs=250):
    print(f'##### {subset_value} #####')
    print(adata.obs[celltype_key].value_counts())

    print(f'filtering features...')
    selected = filter_genes(adata, cell_count_cutoff=5, cell_percentage_cutoff2=0.03, nonz_mean_cutoff=1.12)
    adata = adata[:, selected].copy()

    print(f'removing cell types which occur very seldom...')
    counts = adata.obs[celltype_key].value_counts()
    adata = adata[adata.obs[celltype_key].isin(list(counts.keys()[counts > 2]))]

    print(f'setting up model...')
    RegressionModel.setup_anndata(adata=adata, batch_key='batch_key', labels_key=celltype_key)
    mod = RegressionModel(adata)
    mod.view_anndata_setup()
    
    print(f'training model for {max_epochs} epochs...')
    mod.train(max_epochs=max_epochs, batch_size=2500, train_size=1, lr=0.002, use_gpu=True)
    
    print(f'view history...')
    mod.plot_history(20)
    
    print(f'extract posteriors...')
    adata = mod.export_posterior(adata, sample_kwargs={'num_samples': 1000, 'batch_size': 2500, 'use_gpu': True})
    print(f'saving model to {ref_run_name}/{subset_value}...')
    mod.save(f"{ref_run_name}/{subset_value}/", overwrite=True)
    print(f'saving reference to {ref_run_name}/{subset_value}...')
    adata.write(f"{ref_run_name}/{subset_value}/sc.h5ad")
    print(f'plotting QC...')
    mod.plot_QC()
    
    if 'means_per_cluster_mu_fg' in adata.varm.keys():
        inf_aver = adata.varm['means_per_cluster_mu_fg'][[f'means_per_cluster_mu_fg_{i}' for i in adata.uns['mod']['factor_names']]].copy()
    else:
        inf_aver = adata.var[[f'means_per_cluster_mu_fg_{i}' for i in adata.uns['mod']['factor_names']]].copy()
    inf_aver.columns = adata.uns['mod']['factor_names']
    inf_aver.to_csv(f"{ref_run_name}/{subset_value}/inf_aver.csv")
    
    del adata, mod, inf_aver, selected, counts
```

Example 12 (python):
```python
# Read scRNA data
scRNA_path='/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw.h5ad'
adata = sc.read(scRNA_path)
adata
```

Example 13 (unknown):
```unknown
AnnData object with n_obs × n_vars = 297473 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'mid_grain_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 14 (python):
```python
adata.X.data[:10]
```

Example 15 (unknown):
```unknown
array([ 1.,  1.,  5.,  5.,  8.,  4., 13.,  4.,  3.,  1.], dtype=float32)
```

Example 16 (python):
```python
adata.layers["counts"]=adata.X.copy() # counts to .X
```

Example 17 (python):
```python
adata.X.data[:10]
```

Example 18 (unknown):
```unknown
array([ 1.,  1.,  5.,  5.,  8.,  4., 13.,  4.,  3.,  1.], dtype=float32)
```

Example 19 (python):
```python
# Import custom annotation
fine_grain_modified=pd.read_csv("fine_grain_modified.csv", index_col=0)
fine_grain_modified
```

Example 20 (python):
```python
adata.obs=adata.obs.merge(fine_grain_modified,left_index=True, right_index=True)
adata.obs.head(3)
```

Example 21 (python):
```python
set(adata.obs[celltype_key])
```

Example 22 (unknown):
```unknown
{'AtrialCardiomyocytesCycling',
 'AtrialCardiomyocytesImmature',
 'AtrialCardiomyocytesLeft',
 'AtrialCardiomyocytesRight',
 'AtrioventricularNodeCardiomyocytes',
 'BCells',
 'BCellsMS4A1pos',
 'ChromaffinCells',
 'CoronaryArterialEndothelialCells',
 'CoronaryCapillaryEndothelialCells',
 'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'CoronaryVenousEndothelialCells',
 'CoronaryVesselAdventitialFibroblasts',
 'DendriticCellsMature',
 'DendriticCellsType1',
 'DuctusArteriosusSmoothMuscleCells',
 'EndocardialCells',
 'EndocardialCushionCells',
 'EpicardiumDerivedCells',
 'GreatVesselAdventitialFibroblasts',
 'GreatVesselArterialEndothelialCells',
 'GreatVesselSmoothMuscleCells',
 'GreatVesselVenousEndothelialCells',
 'InnateLymphoidCells',
 'LymphNodeFibroblasticReticularCells',
 'LymphaticEndothelialCells',
 'MacrophagesATF3pos',
 'MacrophagesCX3CR1pos',
 'MacrophagesLYVE1pos',
 'MacrophagesTIMD4pos',
 'MastCells',
 'Megakaryocytes',
 'MesothelialEpicardialCells',
 'MonocyteDerivedCells',
 'Monocytes',
 'MonocytesMPOpos',
 'MyocardialInterstitialFibroblasts1',
 'MyocardialInterstitialFibroblasts2',
 'Myofibroblasts',
 'NaturalKillerCells',
 'NeuronPrecursors',
 'ParaympatheticNeurons',
 'PericardialCells1',
 'PericardialCells2',
 'PericardialCells3',
 'PlasmacytoidDendriticCells',
 'ProBCells',
 'SchwannCellPrecursors',
 'SchwannCells',
 'SinoatrialNodeCardiomyocytes',
 'SympatheticNeurons',
 'TCellsCD4pos',
 'TCellsCD8pos',
 'TregsCD4pos',
 'ValveEndothelialCells',
 'ValveInterstitialCells',
 'VentricularCardiomyocytesCycling',
 'VentricularCardiomyocytesLeftCompact',
 'VentricularCardiomyocytesLeftTrabeculated',
 'VentricularCardiomyocytesRightCompact',
 'VentricularCardiomyocytesRightTrabeculated',
 'VentricularConductionSystemDistal',
 'VentricularConductionSystemProximal'}
```

Example 23 (python):
```python
adata.var.head(3)
```

Example 24 (python):
```python
adata.var=adata.var.set_index("gene_id")
```

Example 25 (python):
```python
adata.var.head(3)
```

Example 26 (python):
```python
v2_probes=pd.read_csv('FFPE_v2_probe_set.csv',skiprows=[0,1,2,3,4])
v2_probes['gene_id_clean'] = v2_probes['gene_id'].apply(lambda x: x.split('_')[-1] if '_' in x else x)
display(v2_probes.head(3))
v2_probe_genes = list(v2_probes[['DEPRECATED_' not in x for x in v2_probes['gene_id']]]['gene_id'])
print(len(v2_probe_genes))
print(v2_probe_genes[:5])
```

Example 27 (unknown):
```unknown
54018
['ENSG00000000003', 'ENSG00000000003', 'ENSG00000000003', 'ENSG00000000005', 'ENSG00000000005']
```

Example 28 (python):
```python
# actually do the subsetting, this is on the shared ensemble IDs
print(f"How many ensembl IDs are shared between the scRNAseq and FFPE Visium? {len(set(adata.var_names).intersection(v2_probe_genes))}")
print(adata.shape)
adata = adata[:,list(set(adata.var_names).intersection(v2_probe_genes))]
print(adata.shape)
```

Example 29 (unknown):
```unknown
How many ensembl IDs are shared between the scRNAseq and FFPE Visium? 18536
(297473, 36601)
(297473, 18536)
```

Example 30 (python):
```python
adata.obs['batch_key'].value_counts()
```

Example 31 (unknown):
```unknown
Hst41_Nuclei_Multiome-v1    104878
Hst33_Nuclei_Multiome-v1     38234
Hst39_Nuclei_Multiome-v1     25386
C99_Cell_5prime-v2           17547
C92_Nuclei_Multiome-v1       13770
C94_Cell_5prime-v2           12985
Hst40_Nuclei_Multiome-v1      9708
C97_Cell_5prime-v2            9702
BRC2251_Cell_3prime-v3        9121
BRC2256_Cell_3prime-v3        7948
Hst36_Nuclei_Multiome-v1      7268
BRC2252_Cell_3prime-v3        6854
C86_Cell_5prime-v2            6376
BRC2260_Cell_3prime-v3        6004
BRC2263_Cell_3prime-v3        5662
BRC2262_Cell_3prime-v3        4129
C87_Nuclei_Multiome-v1        3907
C98_Nuclei_Multiome-v1        2660
C104_Nuclei_Multiome-v1       2490
C85_Nuclei_Multiome-v1        2436
C82_Nuclei_Multiome-v1         408
Name: batch_key, dtype: int64
```

Example 32 (python):
```python
set(adata.obs['batch_key'])
```

Example 33 (unknown):
```unknown
{'BRC2251_Cell_3prime-v3',
 'BRC2252_Cell_3prime-v3',
 'BRC2256_Cell_3prime-v3',
 'BRC2260_Cell_3prime-v3',
 'BRC2262_Cell_3prime-v3',
 'BRC2263_Cell_3prime-v3',
 'C104_Nuclei_Multiome-v1',
 'C82_Nuclei_Multiome-v1',
 'C85_Nuclei_Multiome-v1',
 'C86_Cell_5prime-v2',
 'C87_Nuclei_Multiome-v1',
 'C92_Nuclei_Multiome-v1',
 'C94_Cell_5prime-v2',
 'C97_Cell_5prime-v2',
 'C98_Nuclei_Multiome-v1',
 'C99_Cell_5prime-v2',
 'Hst33_Nuclei_Multiome-v1',
 'Hst36_Nuclei_Multiome-v1',
 'Hst39_Nuclei_Multiome-v1',
 'Hst40_Nuclei_Multiome-v1',
 'Hst41_Nuclei_Multiome-v1'}
```

Example 34 (unknown):
```unknown
subset=False
```

Example 35 (python):
```python
%%time
if subset:
    subset_values=adata.obs[sc_subset_key].unique().tolist()
    for value in subset_values:
        adata_ref_subset = adata[adata.obs[sc_subset_key].isin([value])].copy()
        c2l_make_reference(adata_ref_subset, value)
else:
    adata_ref = adata.copy()
    c2l_make_reference(adata_ref, "all_data")
```

Example 36 (unknown):
```unknown
##### LA #####
AtrialCardiomyocytesLeft                  19443
MP_LYVE1+                                  6955
Pericyte_1                                 5845
EC_capillary                               5764
SAN_FB                                     4311
                                          ...  
valve_FB_MV2                                  5
VentricularCardiomyocytesLeftStressed         2
valve_FB_CS                                   1
VentricularCardiomyocytesRight                1
VentricularCardiomyocytesRightStressed        1
Name: cell_state, Length: 70, dtype: int64
filtering features...
```

Example 37 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pandas/core/arraylike.py:402: RuntimeWarning: divide by zero encountered in log10
  result = getattr(ufunc, method)(*inputs, **kwargs)
```

Example 38 (unknown):
```unknown
removing cell types which occur very seldom...
setting up model...
```

Example 39 (unknown):
```unknown
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
```

Example 40 (unknown):
```unknown
Anndata setup with scvi-tools version 0.20.3.
```

Example 41 (unknown):
```unknown
Setup via `RegressionModel.setup_anndata` with arguments:
```

Example 42 (unknown):
```unknown
{
│   'layer': None,
│   'batch_key': 'batch_key',
│   'labels_key': 'cell_state',
│   'categorical_covariate_keys': None,
│   'continuous_covariate_keys': None
}
```

Example 43 (unknown):
```unknown
Summary Statistics         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃     Summary Stat Key     ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│         n_batch          │  30   │
│         n_cells          │ 89744 │
│ n_extra_categorical_covs │   0   │
│ n_extra_continuous_covs  │   0   │
│         n_labels         │  66   │
│          n_vars          │ 13571 │
└──────────────────────────┴───────┘
```

Example 44 (python):
```python
Data Registry                
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Registry Key ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│      X       │          adata.X          │
│    batch     │ adata.obs['_scvi_batch']  │
│    ind_x     │   adata.obs['_indices']   │
│    labels    │ adata.obs['_scvi_labels'] │
└──────────────┴───────────────────────────┘
```

Example 45 (python):
```python
batch State Registry                           
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃    Source Location     ┃       Categories        ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['batch_key'] │ A61_Nuclei_Multiome-v1  │          0          │
│                        │ AH5_Nuclei_Multiome-v1  │          1          │
│                        │ AV1_Nuclei_Multiome-v1  │          2          │
│                        │ AV2_Nuclei_Multiome-v1  │          3          │
│                        │ AV10_Nuclei_Multiome-v1 │          4          │
│                        │ AV11_Nuclei_Multiome-v1 │          5          │
│                        │ AV13_Nuclei_Multiome-v1 │          6          │
│                        │ AV14_Nuclei_Multiome-v1 │          7          │
│                        │   D1_Nuclei_3prime-v2   │          8          │
│                        │   D2_Nuclei_3prime-v2   │          9          │
│                        │    D3_Cell_3prime-v2    │         10          │
│                        │   D3_Nuclei_3prime-v2   │         11          │
│                        │    D4_Cell_3prime-v2    │         12          │
│                        │   D4_Nuclei_3prime-v2   │         13          │
│                        │    D5_Cell_3prime-v2    │         14          │
│                        │   D5_Nuclei_3prime-v2   │         15          │
│                        │    D6_Cell_3prime-v2    │         16          │
│                        │    D6_Cell_3prime-v3    │         17          │
│                        │   D6_Nuclei_3prime-v2   │         18          │
│                        │    D7_Cell_3prime-v2    │         19          │
│                        │   D7_Nuclei_3prime-v2   │         20          │
│                        │  D8_Nuclei_Multiome-v1  │         21          │
│                        │   D11_Cell_3prime-v3    │         22          │
│                        │  D11_Nuclei_3prime-v3   │         23          │
│                        │   H2_Nuclei_3prime-v3   │         24          │
│                        │   H3_Nuclei_3prime-v3   │         25          │
│                        │   H4_Nuclei_3prime-v3   │         26          │
│                        │   H5_Nuclei_3prime-v3   │         27          │
│                        │   H6_Nuclei_3prime-v3   │         28          │
│                        │   H7_Nuclei_3prime-v3   │         29          │
└────────────────────────┴─────────────────────────┴─────────────────────┘
```

Example 46 (python):
```python
labels State Registry                                
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃     Source Location     ┃            Categories             ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['cell_state'] │            Adipocytes             │          0          │
│                         │       AppendageNeuralCells        │          1          │
│                         │       Artifact mesothelial        │          2          │
│                         │          Atrial pericyte          │          3          │
│                         │     AtrialCardiomyocytesLeft      │          4          │
│                         │     AtrialCardiomyocytesRight     │          5          │
│                         │      AtrialConductionSystem       │          6          │
│                         │                 B                 │          7          │
│                         │             B_plasma              │          8          │
│                         │               CD4+T               │          9          │
│                         │            CD4+T_naive            │         10          │
│                         │             CD4+T_reg             │         11          │
│                         │               CD8+T               │         12          │
│                         │     CD55+ FGFR1+ mesothelial      │         13          │
│                         │        CRIM1+ mesothelial         │         14          │
│                         │           Coronary vSMC           │         15          │
│                         │     CoronaryArteryAdipocytes      │         16          │
│                         │                DC                 │         17          │
│                         │            EC_arterial            │         18          │
│                         │           EC_capillary            │         19          │
│                         │       EC_capillary_IGFBP5+        │         20          │
│                         │        EC_capillary_VEGFA+        │         21          │
│                         │     EC_capillary_VEGFA+NOVA1+     │         22          │
│                         │        EC_capillary_immune        │         23          │
│                         │          EC_endocardial           │         24          │
│                         │             EC_venous             │         25          │
│                         │         EClikeNeuralCells         │         26          │
│                         │         Great vessel vSMC         │         27          │
│                         │     LymphaticEndothelialCells     │         28          │
│                         │             MP_CXCL8+             │         29          │
│                         │             MP_FABP5+             │         30          │
│                         │             MP_LYVE1+             │         31          │
│                         │          MP_LYVE1+AGBL4+          │         32          │
│                         │            MP_cycling             │         33          │
│                         │             MastCells             │         34          │
│                         │        Microvascular SMC_1        │         35          │
│                         │        Microvascular SMC_2        │         36          │
│                         │               Mo/MP               │         37          │
│                         │               Mono                │         38          │
│                         │             NK_CD16hi             │         39          │
│                         │             NK_CD56hi             │         40          │
│                         │            NeuralCells            │         41          │
│                         │         NodalNeuralCells          │         42          │
│                         │          PacemakerCells           │         43          │
│                         │            Pericyte_1             │         44          │
│                         │            Pericyte_2             │         45          │
│                         │            Pericyte_3             │         46          │
│                         │        Proliferating vSMC         │         47          │
│                         │      PulmonaryVeinAdipocytes      │         48          │
│                         │              SAN_FB               │         49          │
│                         │     SOD2+ TIMP1+ mesothelial      │         50          │
│                         │   SinoatrialNodeCardiomyocytes    │         51          │
│                         │   VentricularCardiomyocytesLeft   │         52          │
│                         │ VentricularConductionSystemDistal │         53          │
│                         │         VesselNeuralCells         │         54          │
│                         │               aFB1                │         55          │
│                         │               aFB4                │         56          │
│                         │             doublets              │         57          │
│                         │               vFB1                │         58          │
│                         │          vFB2_activated           │         59          │
│                         │              vSMC_1               │         60          │
│                         │              vSMC_2               │         61          │
│                         │            valve_FB_AV            │         62          │
│                         │           valve_FB_MV1            │         63          │
│                         │           valve_FB_MV2            │         64          │
│                         │            vascular_FB            │         65          │
└─────────────────────────┴───────────────────────────────────┴─────────────────────┘
```

Example 47 (unknown):
```unknown
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
```

Example 48 (unknown):
```unknown
training model for 250 epochs...
```

Example 49 (unknown):
```unknown
You are using a CUDA device ('NVIDIA A100-SXM4-80GB') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
```

Example 50 (python):
```python
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File <timed exec>:5

Cell In[17], line 24, in c2l_make_reference(adata, subset_value, max_epochs)
     21 mod.view_anndata_setup()
     23 print(f'training model for {max_epochs} epochs...')
---> 24 mod.train(max_epochs=max_epochs, batch_size=2500, train_size=1, lr=0.002, use_gpu=True)
     26 print(f'view history...')
     27 mod.plot_history(20)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/cell2location/models/reference/_reference_model.py:157, in RegressionModel.train(self, max_epochs, batch_size, train_size, lr, **kwargs)
    154 kwargs["train_size"] = train_size
    155 kwargs["lr"] = lr
--> 157 super().train(**kwargs)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scvi/model/base/_pyromixin.py:172, in PyroSviTrainMixin.train(self, max_epochs, use_gpu, train_size, validation_size, batch_size, early_stopping, lr, training_plan, plan_kwargs, **trainer_kwargs)
    162 trainer_kwargs["callbacks"].append(PyroJitGuideWarmup())
    164 runner = self._train_runner_cls(
    165     self,
    166     training_plan=training_plan,
   (...)
    170     **trainer_kwargs,
    171 )
--> 172 return runner()

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scvi/train/_trainrunner.py:82, in TrainRunner.__call__(self)
     79 if hasattr(self.data_splitter, "n_val"):
     80     self.training_plan.n_obs_validation = self.data_splitter.n_val
---> 82 self.trainer.fit(self.training_plan, self.data_splitter)
     83 self._update_history()
     85 # data splitter only gets these attrs after fit

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scvi/train/_trainer.py:193, in Trainer.fit(self, *args, **kwargs)
    187 if isinstance(args[0], PyroTrainingPlan):
    188     warnings.filterwarnings(
    189         action="ignore",
    190         category=UserWarning,
    191         message="`LightningModule.configure_optimizers` returned `None`",
    192     )
--> 193 super().fit(*args, **kwargs)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py:608, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    606 model = self._maybe_unwrap_optimized(model)
    607 self.strategy._lightning_module = model
--> 608 call._call_and_handle_interrupt(
    609     self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    610 )

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py:38, in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
     36         return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
     37     else:
---> 38         return trainer_fn(*args, **kwargs)
     40 except _TunerExitException:
     41     trainer._call_teardown_hook()

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py:650, in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    643 ckpt_path = ckpt_path or self.resume_from_checkpoint
    644 self._ckpt_path = self._checkpoint_connector._set_ckpt_path(
    645     self.state.fn,
    646     ckpt_path,  # type: ignore[arg-type]
    647     model_provided=True,
    648     model_connected=self.lightning_module is not None,
    649 )
--> 650 self._run(model, ckpt_path=self.ckpt_path)
    652 assert self.state.stopped
    653 self.training = False

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py:1093, in Trainer._run(self, model, ckpt_path)
   1090 self._logger_connector.reset_metrics()
   1092 # strategy will configure model and move it to the device
-> 1093 self.strategy.setup(self)
   1095 # hook
   1096 if self.state.fn == TrainerFn.FITTING:

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/strategies/single_device.py:73, in SingleDeviceStrategy.setup(self, trainer)
     72 def setup(self, trainer: pl.Trainer) -> None:
---> 73     self.model_to_device()
     74     super().setup(trainer)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/strategies/single_device.py:70, in SingleDeviceStrategy.model_to_device(self)
     68 def model_to_device(self) -> None:
     69     assert self.model is not None, "self.model must be set before self.model.to()"
---> 70     self.model.to(self.root_device)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/lightning_fabric/utilities/device_dtype_mixin.py:54, in _DeviceDtypeModuleMixin.to(self, *args, **kwargs)
     52 device, dtype = torch._C._nn._parse_to(*args, **kwargs)[:2]
     53 self.__update_properties(device=device, dtype=dtype)
---> 54 return super().to(*args, **kwargs)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/modules/module.py:1145, in Module.to(self, *args, **kwargs)
   1141         return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
   1142                     non_blocking, memory_format=convert_to_format)
   1143     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
-> 1145 return self._apply(convert)

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/modules/module.py:797, in Module._apply(self, fn)
    795 def _apply(self, fn):
    796     for module in self.children():
--> 797         module._apply(fn)
    799     def compute_should_use_set_data(tensor, tensor_applied):
    800         if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):
    801             # If the new tensor has compatible tensor type as the existing tensor,
    802             # the current behavior is to change the tensor in-place using `.data =`,
   (...)
    807             # global flag to let the user control whether they want the future
    808             # behavior of overwriting the existing tensor or not.

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/modules/module.py:797, in Module._apply(self, fn)
    795 def _apply(self, fn):
    796     for module in self.children():
--> 797         module._apply(fn)
    799     def compute_should_use_set_data(tensor, tensor_applied):
    800         if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):
    801             # If the new tensor has compatible tensor type as the existing tensor,
    802             # the current behavior is to change the tensor in-place using `.data =`,
   (...)
    807             # global flag to let the user control whether they want the future
    808             # behavior of overwriting the existing tensor or not.

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/modules/module.py:844, in Module._apply(self, fn)
    842 for key, buf in self._buffers.items():
    843     if buf is not None:
--> 844         self._buffers[key] = fn(buf)
    846 return self

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/modules/module.py:1143, in Module.to.<locals>.convert(t)
   1140 if convert_to_format is not None and t.dim() in (4, 5):
   1141     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
   1142                 non_blocking, memory_format=convert_to_format)
-> 1143 return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/Visium_mapping/c2l_pt3_plot_FFPE_immature.html

**Contents:**
- Functions¶
- Read in metadata¶
- Data, parameters¶

2045 rows × 6 columns

**Examples:**

Example 1 (unknown):
```unknown
run='20Mar2024_fetal_FFPE'
```

Example 2 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 3 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Global seed set to 0
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 4 (python):
```python
import gc
import torch
```

Example 5 (python):
```python
def read_and_qc(sample_name, path):
    r""" This function reads the data for one 10X spatial experiment into the anndata object.
    It also calculates QC metrics. Modify this function if required by your workflow.

    :param sample_name: Name of the sample
    :param path: path to data
    """

    adata = sc.read_visium(path + str(sample_name),
                           count_file='filtered_feature_bc_matrix.h5', load_images=True)
    adata.var_names_make_unique(join='-')
    
    adata.obs['sample'] = sample_name
    
    # rename genes to ENSEMBL
    adata.var['SYMBOL'] = adata.var_names
    adata.var_names = adata.var['gene_ids']
    adata.var_names.name = None

    # Calculate QC metrics
    sc.pp.calculate_qc_metrics(adata, inplace=True)
    adata.var['mt'] = [gene.startswith('MT-') for gene in adata.var['SYMBOL']]
    adata.obs['mt_frac'] = adata[:, adata.var['mt'].tolist()].X.sum(1).A.squeeze()/adata.obs['total_counts']

    # add sample name to obs names
    adata.obs["sample"] = [str(i) for i in adata.obs['sample']]
    adata.obs_names = adata.obs["sample"] \
                          + '_' + adata.obs_names
    adata.obs.index.name = 'spot_id'

    return adata

def select_slide(adata, s, s_col='sample'):
    r""" This function selects the data for one slide from the spatial anndata object.

    :param adata: Anndata object with multiple spatial experiments
    :param s: name of selected experiment
    :param s_col: column in adata.obs listing experiment name for each location
    """

    slide = adata[adata.obs[s_col].isin([s]), :]
    s_keys = list(slide.uns['spatial'].keys())
    s_spatial = np.array(s_keys)[[s in k for k in s_keys]][0]

    slide.uns['spatial'] = {s_spatial: slide.uns['spatial'][s_spatial]}

    return slide
```

Example 6 (python):
```python
meta=pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230208_for-cell2loc.csv',
                index_col=0)
meta
```

Example 7 (python):
```python
pd.crosstab(meta['donor'],meta['cell2loc-map_group'])
```

Example 8 (unknown):
```unknown
# select slides
# Healthy FFPE
meta = meta[meta['cell2loc-map_group'].isin(['FFPE_16PCW'])]

sampleIDs = list(meta['sangerID'])
len(sampleIDs)
```

Example 9 (python):
```python
import os
results_folder = os.path.join(os.getcwd(),"analysis/Visium/cell2location_results")

ref_run_name = f'{results_folder}/reference_signatures/{run}'
run_name = f'{results_folder}/cell2location_map/{run}'
```

Example 10 (unknown):
```unknown
visium_path='/nfs/team205/heart/mapped/spaceranger200/'

celltype_key='fine_grain_modified'

vis_subset_key='cell2loc-map_group'
```

Example 11 (python):
```python
%%time
subset_values = os.listdir(os.path.join(results_folder, "cell2location_map", run))
print(f"Found these subsets {subset_values}")

for value in subset_values:
    print(f"###### {value} ######")
    adata_sub = sc.read(f"{run_name}/{value}/sp.h5ad")
    print("visium read in")
    adata_sub.obs[adata_sub.uns['mod']['factor_names']] = adata_sub.obsm['q05_cell_abundance_w_sf']
    IDs = meta[meta[vis_subset_key] == value].sangerID.tolist()
    print(f"{value}: {IDs}")

    for sample in IDs:
        try:
            slide = select_slide(adata_sub, sample)
        except IndexError as e:
            print(f"Sample {sample} not found in adata: {e}")
            continue  # Skip this sample if not found

        slide.obs[slide.uns['mod']['factor_names']] = slide.obsm['q05_cell_abundance_w_sf']
        list_of_cell_types = slide.uns['mod']['factor_names'].tolist()
        print("plotting HE image...")
        sc.pl.spatial(slide, color=None, size=0, img_key='hires', title=f"HE:{sample}", 
                      save=f"{run}__{value}__{sample}__HE.png", show=False)

        # Plotting abundances for all relevant cell states
        if list_of_cell_types:
            print("plotting abundances for all cell states...")
            sc.pl.spatial(slide, color=list_of_cell_types, cmap="RdPu", ncols=10, size=1.3, 
                          img_key='hires', vmin=0, vmax='p99.2', alpha_img=0.3, 
                          save=f"{run}__{value}__{sample}__all_cell_states_abundances.png", show=False)
```

Example 12 (unknown):
```unknown
Found these subsets ['FFPE_16PCW']
###### FFPE_16PCW ######
visium read in
FFPE_16PCW: ['BHF_F_Hea13295830', 'BHF_F_Hea13295831']
```

Example 13 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 14 (unknown):
```unknown
plotting HE image...
```

Example 15 (python):
```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File <timed exec>:22

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:1003, in spatial(adata, basis, img, img_key, library_id, crop_coord, alpha_img, bw, size, scale_factor, spot_size, na_color, show, return_fig, save, **kwargs)
   1000     cmap_img = None
   1001 circle_radius = size * scale_factor * spot_size * 0.5
-> 1003 axs = embedding(
   1004     adata,
   1005     basis=basis,
   1006     scale_factor=scale_factor,
   1007     size=circle_radius,
   1008     na_color=na_color,
   1009     show=False,
   1010     save=False,
   1011     **kwargs,
   1012 )
   1013 if not isinstance(axs, list):
   1014     axs = [axs]

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392, in embedding(adata, basis, color, gene_symbols, use_raw, sort_order, edges, edges_width, edges_color, neighbors_key, arrows, arrows_kwds, groups, components, dimensions, layer, projection, scale_factor, color_map, cmap, palette, na_color, na_in_legend, size, frameon, legend_fontsize, legend_fontweight, legend_loc, legend_fontoutline, colorbar_loc, vmax, vmin, vcenter, norm, add_outline, outline_width, outline_color, ncols, hspace, wspace, title, show, save, ax, return_fig, **kwargs)
    389         # if user did not set alpha, set alpha to 0.7
    390         kwargs['alpha'] = 0.7 if alpha is None else alpha
--> 392     cax = scatter(
    393         coords[:, 0],
    394         coords[:, 1],
    395         marker=".",
    396         c=color_vector,
    397         rasterized=settings._vector_friendly,
    398         norm=normalize,
    399         **kwargs,
    400     )
    402 # remove y and x ticks
    403 ax.set_yticks([])

File /lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_utils.py:1108, in circles(x, y, s, ax, marker, c, vmin, vmax, scale_factor, **kwargs)
   1105 # You can set `facecolor` with an array for each patch,
   1106 # while you can only set `facecolors` with a value for all.
   1107 if scale_factor != 1.0:
-> 1108     x = x * scale_factor
   1109     y = y * scale_factor
   1110 zipped = np.broadcast(x, y, s)

TypeError: can't multiply sequence by non-int of type 'float'
```

Example 16 (python):
```python
%%time

adata_dict={}

subset_values = os.listdir(os.path.join(results_folder, "cell2location_map", run))
print(f"Found these subsets {subset_values}")

for value in subset_values:
    print(f"###### {value} ######")
    adata_sub = sc.read(f"{run_name}/{value}/sp.h5ad")
    print("visium read in")
    adata_sub.obs[adata_sub.uns['mod']['factor_names']] = adata_sub.obsm['q05_cell_abundance_w_sf']
    IDs = meta[meta[vis_subset_key] == value].sangerID.tolist()
    print(f"{value}: {IDs}")

    for sample in IDs:
        try:
            slide = select_slide(adata_sub, sample)
        except IndexError as e:
            print(f"Sample {sample} not found in adata: {e}")
            continue  # Skip this sample if not found

        slide.obs[slide.uns['mod']['factor_names']] = slide.obsm['q05_cell_abundance_w_sf']
        adata_dict[sample]=slide
```

Example 17 (unknown):
```unknown
Found these subsets ['FFPE_16PCW']
###### FFPE_16PCW ######
visium read in
FFPE_16PCW: ['BHF_F_Hea13295830', 'BHF_F_Hea13295831']
```

Example 18 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 19 (unknown):
```unknown
CPU times: user 793 ms, sys: 1.66 s, total: 2.45 s
Wall time: 3.88 s
```

Example 20 (python):
```python
adata_dict.keys()
```

Example 21 (unknown):
```unknown
dict_keys(['BHF_F_Hea13295830', 'BHF_F_Hea13295831'])
```

Example 22 (python):
```python
from tqdm import tqdm
atrial_data_dict={}
for sample,tmp_adata in tqdm(adata_dict.items()):
    print(f"sample: {sample},  age is {tmp_adata.obs.age.unique().tolist()[0]}")
    tmp_df=tmp_adata[tmp_adata.obs.JC_edit=="Myocardium - Atrial"].obs[['AtrialCardiomyocytesLeft_immature', 'AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight','AtrialCardiomyocytesRight_immature']]

    if tmp_df.shape[0]>0:
        tmp_df["sample"]=sample
        tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
        atrial_data_dict[sample]=tmp_df
    else:
        print(f"no atrial myocardium spots annotated for sample: {sample}")
```

Example 23 (python):
```python
0%|                                                                                                                                                                                                                     | 0/2 [00:00<?, ?it/s]/tmp/ipykernel_266704/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_266704/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_266704/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_266704/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 133.30it/s]
```

Example 24 (unknown):
```unknown
sample: BHF_F_Hea13295830,  age is PCW-16w
sample: BHF_F_Hea13295831,  age is PCW-16w
```

Example 25 (python):
```python
data=pd.concat(list(atrial_data_dict.values()))
data
```

Example 26 (unknown):
```unknown
data.to_csv("AtrialProportionsFFPE.csv")
```

Example 27 (unknown):
```unknown
# Assuming 'data' is your pandas DataFrame
# Calculate the total for each row
total = data['AtrialCardiomyocytesImmature'] + data['AtrialCardiomyocytesRight'] + data['AtrialCardiomyocytesLeft']

# Calculate the proportions for each column
data['AtrialCardiomyocytesImmature_prop'] = data['AtrialCardiomyocytesImmature'] / total
data['AtrialCardiomyocytesRight_prop'] = data['AtrialCardiomyocytesRight'] / total
data['AtrialCardiomyocytesLeft_prop'] = data['AtrialCardiomyocytesLeft'] / total

# The new DataFrame 'data' now includes the proportion columns
data
```

Example 28 (unknown):
```unknown
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[17], line 3
      1 # Assuming 'data' is your pandas DataFrame
      2 # Calculate the total for each row
----> 3 total = data['AtrialCardiomyocytesImmature'] + data['AtrialCardiomyocytesRight'] + data['AtrialCardiomyocytesLeft']
      5 # Calculate the proportions for each column
      6 data['AtrialCardiomyocytesImmature_prop'] = data['AtrialCardiomyocytesImmature'] / total

NameError: name 'data' is not defined
```

Example 29 (python):
```python
import plotly.express as px
fig = px.scatter_ternary(data,
                         a="AtrialCardiomyocytesImmature_prop", b="AtrialCardiomyocytesLeft_prop", c="AtrialCardiomyocytesRight_prop",
                        color="age",
                        opacity=0.8)
fig.show()
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/CellOracle/CellOracle_no1_construct-GRN_Left-cardiomyocytes.html

**Contents:**
- Import libraries¶
- Load gene-expression data¶
- Make oracle object¶
- Add scenicplus base GRN¶
- KNN imputation¶
- GRN calculation¶
- Network preprocessing for perturbation simulation¶
- Save the oracle object¶
- Network preprocessing¶
- Network analysis; Network score for each gene¶
- Find parameters for n_grid and min_mass for perturbation simulation¶
- Make predictive modelos for simulation¶
- In silico TF perturbation analysis¶
- Visualisation¶
- Find parameters for n_grid and min_mass¶
- Plot vector field¶
- Compare simulation vector with development vectors¶
- Focus on a single development lineage to interpret the results in detail¶

ref: https://morris-lab.github.io/CellOracle.documentation/notebooks/04_Network_analysis/Network_analysis_with_Paul_etal_2015_data.html

CellOracle uses the same strategy as velocyto for visualizing cell transitions. This process requires KNN imputation in advance.

For the KNN imputation, we first need to calculate and select PCs.

Estimate the optimal number of nearest neighbors for KNN imputation.

67559 rows × 6 columns

from below ref: https://morris-lab.github.io/CellOracle.documentation/notebooks/05_simulation/Gata1_KO_simulation_with_Paul_etal_2015_data.html

Here, we will need to fit the ridge regression models again. This process will take less time than the GRN inference in the previous notebook, because we are using the filtered GRN models.

The next step calculates the probability of cell state transitions based on the simulation data. You can use the transition probabilities between cells to predict how cells will change after a perturbation.

This transition probability will be used later.

Caution: It is very important to find the optimal scale parameter.

If the vectors are not visible, you can try a smaller scale parameter to magnify the vector length. However, if you see large vectors in the randomized results (right panel), it means that the scale parameter is too small.

Again, we need to adjust the scale parameter. Please seek to find the optimal scale parameter that provides good visualization.

If you don’t see any vector, you can try the smaller scale parameter to magnify the vector length. However, if you see large vectors in the randomized results (right panel), it means the scale parameter is too small.

Here, we will calculate the development vector field using pseudotime gradient as follows.

Note: Other methods may be used to represent a continuous scRNA-seq trajectory flow. For example, RNA velocity analysis is a good way to estimate the direction of cell differentiation. Choose the method that best suits your data.

In the analysis below, we will use pseudotime data. Pseudotime data is included in the demo data. If you are analyzing your own scRNA-seq data, please calculate pseudotime before starting this analysis.

Next, we convert the pseudotime data into grid points. For this calculation we can chose one of two methods.

knn: K-Nearesr Neighbor regression. You will need to set number of neighbors. Please adjust n_knn for best results.This will depend on the population size and density of your scRNA-seq data.

gradient.transfer_data_into_grid(args={"method": "knn", "n_knn":50})

polynomial: Polynomial regression using x-axis and y-axis of dimensional reduction space.

In general, this method will be more robust. Please use this method if knn method does not work. n_poly is the number of degrees for the polynomial regression model. Please try to find appropriaten_poly searching for best results.

gradient.transfer_data_into_grid(args={"method": "polynomial", "n_poly":3})

Calculate the 2D vector map to represents the pseudotime gradient. After the gradient calculation, the length of the vector will be normalized automatically.

Please adjust the scale parameter to adjust vector length visualization.

The inner product represents the similarity between two vectors.

Using the inner product, we compare the 2D vector field of perturbation simulation and development flow.

The inner product will be a positive value when two vectors are pointing in the same direction.

Conversely, the inner product will be a negative value when two vectors are pointing in the opposing directions.

The length of the vectors also affects the magnitude of inner product value.

In summary, we quantitatively compare the directionality and size of vectors between perturbation simulation and natural differentiation using inner product, and we define the score as perturbation score (PS).

A negative PS means that the TF perturbation would block differentiation.

A positive PS means that the TF perturbation would promote differentiation.

Here, need to adjust the vm parameter for the PS score color visualization. Please seek to find the optimal vm parameter that provides good visualization.

If you don’t see any color in the left panel below, you can try the smaller vm parameter to magnify the scale of vm visualization. However, if you see colors in the randomized results (right panel), it means the vm parameter is too small.

**Examples:**

Example 1 (python):
```python
import os
import sys

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scanpy as sc
import seaborn as sns
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.6.3
numpy               1.23.5
pandas              1.5.3
scanpy              1.9.2
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                 9.4.0
anndata             0.8.0
asttokens           NA
backcall            0.2.0
beta_ufunc          NA
binom_ufunc         NA
cffi                1.15.1
colorama            0.4.6
comm                0.1.2
cycler              0.10.0
cython_runtime      NA
dateutil            2.8.2
debugpy             1.6.6
decorator           5.1.1
defusedxml          0.7.1
executing           1.2.0
google              NA
h5py                3.8.0
hypergeom_ufunc     NA
igraph              0.10.4
importlib_metadata  NA
invgauss_ufunc      NA
ipykernel           6.21.2
ipython_genutils    0.2.0
ipywidgets          8.0.4
jedi                0.18.2
joblib              1.2.0
kiwisolver          1.4.4
llvmlite            0.39.1
louvain             0.8.0
mpl_toolkits        NA
natsort             8.2.0
nbinom_ufunc        NA
ncf_ufunc           NA
nct_ufunc           NA
ncx2_ufunc          NA
numba               0.56.4
packaging           23.0
parso               0.8.3
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
platformdirs        3.0.0
prompt_toolkit      3.0.36
psutil              5.9.4
ptyprocess          0.7.0
pure_eval           0.2.2
pyarrow             11.0.0
pydev_ipython       NA
pydevconsole        NA
pydevd              2.9.5
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.14.0
pyparsing           3.0.9
pytz                2022.7.1
scipy               1.10.1
setuptools          67.4.0
six                 1.16.0
skewnorm_ufunc      NA
sklearn             1.2.1
stack_data          0.6.2
statsmodels         0.13.5
texttable           1.6.7
threadpoolctl       3.1.0
tornado             6.2
traitlets           5.9.0
typing_extensions   NA
wcwidth             0.2.6
yaml                6.0
zipp                NA
zmq                 25.0.0
```

Example 5 (unknown):
```unknown
-----
IPython             8.4.0
jupyter_client      8.0.3
jupyter_core        5.2.0
notebook            6.5.2
-----
Python 3.8.16 | packaged by conda-forge | (default, Feb  1 2023, 16:01:55) [GCC 11.3.0]
Linux-4.15.0-213-generic-x86_64-with-glibc2.10
-----
Session information updated at 2023-10-17 13:23
```

Example 6 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 7 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 8 (python):
```python
# read in adata
# whole data
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.h5ad')

# subset gene expression data
adata = adata[:,adata.var['modality']=='Gene Expression']

# keep raw cont data before log normalisation
adata.raw = adata
adata.layers["raw_count"] = adata.raw.X.copy()

# filter and log-normalise
sc.pp.filter_genes(adata, min_cells=3)
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

print(adata.X.data[:10])
adata
```

Example 9 (unknown):
```unknown
[1.0619993 1.5653387 1.8985853 1.5653387 2.347629  1.8985853 1.5653387
 1.0619993 1.0619993 1.0619993]
```

Example 10 (unknown):
```unknown
AnnData object with n_obs × n_vars = 16451 × 26187
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N', 'n_cells'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'umap', 'week_colors', 'log1p'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    layers: 'raw_count'
    obsp: 'connectivities', 'distances'
```

Example 11 (python):
```python
# select genes which are targeted or the regulators in the regulons
# 
ereg_signatures = utils.read_pkl("/nfs/team205/heart/pyscenic/plus/eRegulon_signatures/vCM_onlyLeft_Gene_eRegulon_signatures.pkl")
genes_sel = []
for key,value in ereg_signatures.items():
    genes_sel = genes_sel + value
    genes_sel = genes_sel + [key.split('_')[0]]
genes_sel = list(set(genes_sel))

# hvg and targeted
# hvg_targeted = list(set(adata.var_names[adata.var.highly_variable]).intersection(genes_targeted))
# len(hvg_targeted)
len(genes_sel)
```

Example 12 (python):
```python
# subset genes
# adata = adata[:,hvg_targeted]
adata = adata[:,genes_sel]
adata
```

Example 13 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 16451 × 11229
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N', 'n_cells'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'umap', 'week_colors', 'log1p'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    layers: 'raw_count'
    obsp: 'connectivities', 'distances'
```

Example 14 (python):
```python
print(f"Cell number is :{adata.shape[0]}")
print(f"Gene number is :{adata.shape[1]}")
```

Example 15 (unknown):
```unknown
Cell number is :16451
Gene number is :11229
```

Example 16 (python):
```python
# add t_binned category
adata.obs['t_binned']=np.nan

# compact
adata_compact =  sc.read_h5ad('/nfs/team205/heart/anndata_objects/Vitro/post_multiVI_full-features/Foetal_vCM_Left-compact_rna.h5ad',
                              backed='r')
adata.obs.loc[adata_compact.obs_names, 't_binned'] = adata_compact.obs['t_binned'].copy()

# trabeculated
adata_trab =  sc.read_h5ad('/nfs/team205/heart/anndata_objects/Vitro/post_multiVI_full-features/Foetal_vCM_Left-trabeculated_rna.h5ad',
                              backed='r')
adata.obs.loc[adata_trab.obs_names, 't_binned'] = adata_trab.obs['t_binned'].copy()

adata.obs['t_binned'].value_counts()

# combine fine_grain and t_binned category
adata.obs['fine_grain_time-binned'] = adata.obs['fine_grain'].astype('str')+'_'+adata.obs['t_binned'].astype('str')
```

Example 17 (python):
```python
/tmp/ipykernel_900/724224780.py:2: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
  adata.obs['t_binned']=np.nan
```

Example 18 (python):
```python
set(adata.obs['t_binned'])
```

Example 19 (unknown):
```unknown
{'t1', 't2', 't3'}
```

Example 20 (python):
```python
sc.pl.draw_graph(adata, color=['fine_grain','t','fine_grain_time-binned'], size=5, wspace=0.3, ncols=3)
```

Example 21 (unknown):
```unknown
/home/jovyan/my-conda-envs/celloracle_env/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/home/jovyan/my-conda-envs/celloracle_env/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 22 (python):
```python
import celloracle as co
co.__version__
```

Example 23 (unknown):
```unknown
# Instantiate Oracle object
oracle = co.Oracle()
```

Example 24 (python):
```python
# Check data in anndata
print("Metadata columns :", list(adata.obs.columns))
print("Dimensional reduction: ", list(adata.obsm.keys()))
```

Example 25 (python):
```python
Metadata columns : ['sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont', 't_binned', 'fine_grain_time-binned']
Dimensional reduction:  ['MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs']
```

Example 26 (python):
```python
# In this notebook, we use the unscaled mRNA count for the nput of Oracle object.
adata.X = adata.layers["raw_count"].copy()

# Instantiate Oracle object.
oracle.import_anndata_as_raw_count(adata=adata,
                                   cluster_column_name="fine_grain_time-binned",
                                   embedding_name="X_draw_graph_fa")
```

Example 27 (python):
```python
11229 genes were found in the adata. Note that Celloracle is intended to use around 1000-3000 genes, so the behavior with this number of genes may differ from what is expected.
WARNING: adata.X seems to be already log-transformed.
```

Example 28 (unknown):
```unknown
# read in scenicplus regulons
regulons = utils.read_pkl('/nfs/team205/heart/pyscenic/plus/baseGRN_forCO/vCM_onlyLeft_baseGRN_act.pkl')
TG_to_TF_dictionary = co.utility.inverse_dictionary(regulons)
```

Example 29 (unknown):
```unknown
0%|          | 0/11200 [00:00<?, ?it/s]
```

Example 30 (unknown):
```unknown
# add
oracle.import_TF_data(TFdict=TG_to_TF_dictionary)
```

Example 31 (unknown):
```unknown
TG_to_TF_dictionary['SGSM2']
```

Example 32 (unknown):
```unknown
['CREB1',
 'HAND2',
 'MEF2A',
 'PLAG1',
 'PRDM16',
 'SREBF2',
 'TEAD4',
 'ZNF148',
 'ZNF316',
 'ZNF576']
```

Example 33 (python):
```python
%%time
%matplotlib inline
# Perform PCA
oracle.perform_PCA()

# Select important PCs
plt.plot(np.cumsum(oracle.pca.explained_variance_ratio_)[:100])
n_comps = np.where(np.diff(np.diff(np.cumsum(oracle.pca.explained_variance_ratio_))>0.002))[0][0]
plt.axvline(n_comps, c="k")
plt.show()
print(n_comps)
n_comps = min(n_comps, 50)
```

Example 34 (unknown):
```unknown
6
CPU times: user 1h 11min 45s, sys: 15min 10s, total: 1h 26min 56s
Wall time: 8min 54s
```

Example 35 (python):
```python
oracle.pcs.shape
```

Example 36 (unknown):
```unknown
(16451, 11229)
```

Example 37 (python):
```python
n_cell = oracle.adata.shape[0]
print(f"cell number is :{n_cell}")
```

Example 38 (unknown):
```unknown
cell number is :16451
```

Example 39 (python):
```python
k = int(0.025*n_cell)
print(f"Auto-selected k is :{k}")
```

Example 40 (unknown):
```unknown
Auto-selected k is :411
```

Example 41 (unknown):
```unknown
%%time
oracle.knn_imputation(n_pca_dims=n_comps, k=k, balanced=True, b_sight=k*8,
                      b_maxl=k*4, n_jobs=4)
```

Example 42 (unknown):
```unknown
CPU times: user 4min 13s, sys: 5.82 s, total: 4min 18s
Wall time: 4min 4s
```

Example 43 (unknown):
```unknown
# Save oracle object.
oracle.to_hdf5("/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.celloracle.oracle")
```

Example 44 (unknown):
```unknown
# Load
oracle = co.load_hdf5("/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.celloracle.oracle")
```

Example 45 (python):
```python
%matplotlib inline
# Check clustering data
sc.pl.draw_graph(oracle.adata, color='fine_grain_time-binned')
```

Example 46 (unknown):
```unknown
%%time
# Calculate GRN for each population in "louvain_annot" clustering unit.
# This step may take some time.(~30 minutes)
links = oracle.get_links(cluster_name_for_GRN_unit='fine_grain_time-binned', alpha=10,
                         verbose_level=10)
```

Example 47 (unknown):
```unknown
0%|          | 0/6 [00:00<?, ?it/s]
```

Example 48 (unknown):
```unknown
Inferring GRN for VentricularCardiomyocytesLeftCompact_t1...
```

Example 49 (unknown):
```unknown
0%|          | 0/11200 [00:00<?, ?it/s]
```

Example 50 (unknown):
```unknown
Inferring GRN for VentricularCardiomyocytesLeftCompact_t2...
```

Example 51 (unknown):
```unknown
0%|          | 0/11200 [00:00<?, ?it/s]
```

Example 52 (unknown):
```unknown
Inferring GRN for VentricularCardiomyocytesLeftCompact_t3...
```

Example 53 (unknown):
```unknown
0%|          | 0/11200 [00:00<?, ?it/s]
```

Example 54 (unknown):
```unknown
Inferring GRN for VentricularCardiomyocytesLeftTrabeculated_t1...
```

Example 55 (unknown):
```unknown
0%|          | 0/11200 [00:00<?, ?it/s]
```

Example 56 (unknown):
```unknown
Inferring GRN for VentricularCardiomyocytesLeftTrabeculated_t2...
```

Example 57 (unknown):
```unknown
0%|          | 0/11200 [00:00<?, ?it/s]
```

Example 58 (unknown):
```unknown
Inferring GRN for VentricularCardiomyocytesLeftTrabeculated_t3...
```

Example 59 (unknown):
```unknown
0%|          | 0/11200 [00:00<?, ?it/s]
```

Example 60 (unknown):
```unknown
CPU times: user 29min 21s, sys: 1min 38s, total: 30min 59s
Wall time: 36min 53s
```

Example 61 (unknown):
```unknown
links.links_dict.keys()
```

Example 62 (unknown):
```unknown
dict_keys(['VentricularCardiomyocytesLeftCompact_t1', 'VentricularCardiomyocytesLeftCompact_t2', 'VentricularCardiomyocytesLeftCompact_t3', 'VentricularCardiomyocytesLeftTrabeculated_t1', 'VentricularCardiomyocytesLeftTrabeculated_t2', 'VentricularCardiomyocytesLeftTrabeculated_t3'])
```

Example 63 (unknown):
```unknown
links.links_dict['VentricularCardiomyocytesLeftCompact_t3']
```

Example 64 (unknown):
```unknown
# Save Links object.
links.to_hdf5(file_path="/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/vCM_onlyLeft_links.celloracle.links")
```

Example 65 (unknown):
```unknown
# load the saved links
links = co.load_hdf5("/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/vCM_onlyLeft_links.celloracle.links")
```

Example 66 (unknown):
```unknown
%%time

links.filter_links(p=0.001, weight="coef_abs", threshold_number=10000)

# Calculate network scores.
links.get_network_score()
display(links.merged_score.head())

# Make predictive modelos for simulation
# Here, we will need to fit the ridge regression models again.
# This process will take less time than the GRN inference in the previous notebook, because we are using the filtered GRN models.
oracle.get_cluster_specific_TFdict_from_Links(links_object=links)
oracle.fit_GRN_for_simulation(alpha=10,
                              use_cluster_specific_TFdict=True)
```

Example 67 (unknown):
```unknown
0%|          | 0/6 [00:00<?, ?it/s]
```

Example 68 (unknown):
```unknown
CPU times: user 16min 9s, sys: 16min 33s, total: 32min 42s
Wall time: 5min 24s
```

Example 69 (unknown):
```unknown
Oracle object

Meta data
    celloracle version used for instantiation: 0.11.0
    n_cells: 16451
    n_genes: 11229
    cluster_name: fine_grain_time-binned
    dimensional_reduction_name: X_draw_graph_fa
    n_target_genes_in_TFdict: 11200 genes
    n_regulatory_in_TFdict: 236 genes
    n_regulatory_in_both_TFdict_and_scRNA-seq: 236 genes
    n_target_genes_both_TFdict_and_scRNA-seq: 11200 genes
    k_for_knn_imputation: 411
Status
    Gene expression matrix: Ready
    BaseGRN: Ready
    PCA calculation: Done
    Knn imputation: Done
    GRN calculation for simulation: Done
```

Example 70 (unknown):
```unknown
oracle.to_hdf5("/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.celloracle.oracle")
```

Example 71 (unknown):
```unknown
links.filter_links(p=0.001, weight="coef_abs", threshold_number=10000)
```

Example 72 (python):
```python
plt.rcParams["figure.figsize"] = [7, 3]
```

Example 73 (unknown):
```unknown
links.plot_degree_distributions(plot_model=True,
                                               #save=f"{save_folder}/degree_distribution/",
                                               )
```

Example 74 (unknown):
```unknown
VentricularCardiomyocytesLeftCompact_t1
```

Example 75 (unknown):
```unknown
VentricularCardiomyocytesLeftCompact_t2
```

Example 76 (unknown):
```unknown
VentricularCardiomyocytesLeftCompact_t3
```

Example 77 (unknown):
```unknown
VentricularCardiomyocytesLeftTrabeculated_t1
```

Example 78 (unknown):
```unknown
VentricularCardiomyocytesLeftTrabeculated_t2
```

Example 79 (unknown):
```unknown
VentricularCardiomyocytesLeftTrabeculated_t3
```

Example 80 (unknown):
```unknown
# Calculate network scores.
links.get_network_score()
```

Example 81 (unknown):
```unknown
links.merged_score.head()
```

Example 82 (unknown):
```unknown
# Check cluster name
links.cluster
```

Example 83 (unknown):
```unknown
['VentricularCardiomyocytesLeftCompact_t1',
 'VentricularCardiomyocytesLeftCompact_t2',
 'VentricularCardiomyocytesLeftCompact_t3',
 'VentricularCardiomyocytesLeftTrabeculated_t1',
 'VentricularCardiomyocytesLeftTrabeculated_t2',
 'VentricularCardiomyocytesLeftTrabeculated_t3']
```

Example 84 (python):
```python
# Visualize top n-th genes with high scores.
plt.rcParams["figure.figsize"] = [5, 7]
links.plot_scores_as_rank(cluster="VentricularCardiomyocytesLeftCompact_t3", n_gene=30, save=None)
```

Example 85 (python):
```python
# Compare GRN score between two clusters
plt.rcParams["figure.figsize"] = [5, 5]
links.plot_score_comparison_2D(value="betweenness_centrality",
                               cluster1='VentricularCardiomyocytesLeftCompact_t3', cluster2='VentricularCardiomyocytesLeftCompact_t1',
                               percentile=98,
                               save=None)
```

Example 86 (unknown):
```unknown
links.plot_score_comparison_2D(value="betweenness_centrality",
                               cluster1='VentricularCardiomyocytesLeftCompact_t3', cluster2='VentricularCardiomyocytesLeftTrabeculated_t3',
                               percentile=98,
                               save=None)
```

Example 87 (unknown):
```unknown
# set a gene for this trial
goi = "TBX20"
# Enter perturbation conditions to simulate signal propagation after the perturbation.
oracle.simulate_shift(perturb_condition={goi: 0.0}, # this is KO simulation so expression=0
                      n_propagation=3)
```

Example 88 (unknown):
```unknown
%%time
# Get transition probability
oracle.estimate_transition_prob(n_neighbors=200,
                                knn_random=True,
                                sampled_fraction=1)

# Calculate embedding
oracle.calculate_embedding_shift(sigma_corr=0.05)
```

Example 89 (python):
```python
# visualisation
fig, ax = plt.subplots(1, 2,  figsize=[13, 6])

scale = 20
# Show quiver plot
oracle.plot_quiver(scale=scale, ax=ax[0])
ax[0].set_title(f"Simulated cell identity shift vector: {goi} KO")

# Show quiver plot that was calculated with randomized graph.
oracle.plot_quiver_random(scale=scale, ax=ax[1])
ax[1].set_title(f"Randomized simulation vector")

plt.show()
```

Example 90 (unknown):
```unknown
# n_grid = 40 is a good starting value.
n_grid = 40
oracle.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=200)
```

Example 91 (unknown):
```unknown
# Search for best min_mass.
oracle.suggest_mass_thresholds(n_suggestion=12)
```

Example 92 (unknown):
```unknown
min_mass = 0.011
oracle.calculate_mass_filter(min_mass=min_mass, plot=True)
```

Example 93 (unknown):
```unknown
%%time
links.filter_links()
oracle.get_cluster_specific_TFdict_from_Links(links_object=links)
oracle.fit_GRN_for_simulation(alpha=10,
                              use_cluster_specific_TFdict=True)
```

Example 94 (python):
```python
# Check gene expression
goi = "TBX20"
sc.pl.draw_graph(oracle.adata, color=[goi, oracle.cluster_column_name],
                 layer="imputed_count", use_raw=False, cmap="viridis")
```

Example 95 (python):
```python
# Plot gene expression in histogram
sc.get.obs_df(oracle.adata, keys=[goi], layer="imputed_count").hist()
plt.show()
```

Example 96 (unknown):
```unknown
# Enter perturbation conditions to simulate signal propagation after the perturbation.
oracle.simulate_shift(perturb_condition={goi: 0.0}, # this is KO simulation so expression=0
                      n_propagation=3)
```

Example 97 (unknown):
```unknown
%%time
# Get transition probability
oracle.estimate_transition_prob(n_neighbors=200,
                                knn_random=True,
                                sampled_fraction=1)

# Calculate embedding
oracle.calculate_embedding_shift(sigma_corr=0.05)
```

Example 98 (python):
```python
fig, ax = plt.subplots(1, 2,  figsize=[13, 6])

scale = 20
# Show quiver plot
oracle.plot_quiver(scale=scale, ax=ax[0])
ax[0].set_title(f"Simulated cell identity shift vector: {goi} KO")

# Show quiver plot that was calculated with randomized graph.
oracle.plot_quiver_random(scale=scale, ax=ax[1])
ax[1].set_title(f"Randomized simulation vector")

plt.show()
```

Example 99 (unknown):
```unknown
# n_grid = 40 is a good starting value.
n_grid = 40
oracle.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=200)
```

Example 100 (unknown):
```unknown
# Search for best min_mass.
oracle.suggest_mass_thresholds(n_suggestion=12)
```

Example 101 (unknown):
```unknown
min_mass = 0.011
oracle.calculate_mass_filter(min_mass=min_mass, plot=True)
```

Example 102 (python):
```python
fig, ax = plt.subplots(1, 2,  figsize=[13, 6])

scale_simulation = 0.1
# Show quiver plot
oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax[0])
ax[0].set_title(f"Simulated cell identity shift vector: {goi} KO")

# Show quiver plot that was calculated with randomized graph.
oracle.plot_simulation_flow_random_on_grid(scale=scale_simulation, ax=ax[1])
ax[1].set_title(f"Randomized simulation vector")

plt.show()
```

Example 103 (python):
```python
# Plot vector field with cell cluster
fig, ax = plt.subplots(figsize=[8, 8])

oracle.plot_cluster_whole(ax=ax, s=10)
oracle.plot_simulation_flow_on_grid(scale=scale_simulation, ax=ax, show_background=False)
```

Example 104 (python):
```python
# Visualize pseudotime
fig, ax = plt.subplots(figsize=[6,6])

sc.pl.embedding(adata=oracle.adata, basis=oracle.embedding_name, ax=ax, cmap="rainbow",
                color=["t"])
```

Example 105 (python):
```python
from celloracle.applications import Gradient_calculator

# Instantiate Gradient calculator object
gradient = Gradient_calculator(oracle_object=oracle, pseudotime_key="t")
```

Example 106 (unknown):
```unknown
gradient.calculate_p_mass(smooth=0.8, n_grid=n_grid, n_neighbors=200)
gradient.calculate_mass_filter(min_mass=min_mass, plot=True)
```

Example 107 (unknown):
```unknown
gradient.transfer_data_into_grid(args={"method": "knn", "n_knn":50})
```

Example 108 (unknown):
```unknown
# Calculate graddient
gradient.calculate_gradient()

# Show results
scale_dev = 40
gradient.visualize_results(scale=scale_dev, s=5)
```

Example 109 (python):
```python
# Visualize results
fig, ax = plt.subplots(figsize=[6, 6])
gradient.plot_dev_flow_on_grid(scale=scale_dev, ax=ax)
```

Example 110 (python):
```python
from celloracle.applications import Oracle_development_module

# Make Oracle_development_module to compare two vector field
dev = Oracle_development_module()

# Load development flow
dev.load_differentiation_reference_data(gradient_object=gradient)

# Load simulation result
dev.load_perturb_simulation_data(oracle_object=oracle)


# Calculate inner produc scores
dev.calculate_inner_product()
dev.calculate_digitized_ip(n_bins=10)
```

Example 111 (python):
```python
# Show perturbation scores
vm = 0.01

fig, ax = plt.subplots(1, 2, figsize=[12, 6])
dev.plot_inner_product_on_grid(vm=vm, s=50, ax=ax[0])
ax[0].set_title(f"PS")

dev.plot_inner_product_random_on_grid(vm=vm, s=50, ax=ax[1])
ax[1].set_title(f"PS calculated with Randomized simulation vector")
plt.show()
```

Example 112 (python):
```python
# Show perturbation scores with perturbation simulation vector field
fig, ax = plt.subplots(figsize=[6, 6])
dev.plot_inner_product_on_grid(vm=vm, s=50, ax=ax)
dev.plot_simulation_flow_on_grid(scale=scale_simulation, show_background=False, ax=ax)
```

Example 113 (unknown):
```unknown
# Let's visualize the results
dev.visualize_development_module_layout_0(s=5,
                                          scale_for_simulation=scale_simulation,
                                          s_grid=50,
                                          scale_for_pseudotime=scale_dev,
                                          vm=vm)
```

Example 114 (unknown):
```unknown
dev.inner_product_df
```

Example 115 (unknown):
```unknown
dev.cell_idx_use
```

Example 116 (python):
```python
# Get cell index list for the cells of interest
clusters = ['VentricularCardiomyocytesLeftCompact_t1',
 'VentricularCardiomyocytesLeftCompact_t2',
 'VentricularCardiomyocytesLeftCompact_t3']
cell_idx = np.where(oracle.adata.obs["fine_grain_time-binned"].isin(clusters))[0]

# Check index
print(cell_idx)
```

Example 117 (unknown):
```unknown
dev = Oracle_development_module()

# Load development flow
dev.load_differentiation_reference_data(gradient_object=gradient)

# Load simulation result
dev.load_perturb_simulation_data(oracle_object=oracle,
                                 cell_idx_use=cell_idx, # Enter cell id list
                                 name="Lineage_Compact" # Name of this cell group. You can enter any name.
                                 )

# Calculation
dev.calculate_inner_product()
dev.calculate_digitized_ip(n_bins=10)
```

Example 118 (unknown):
```unknown
# Let's visualize the results
dev.visualize_development_module_layout_0(s=5,
                                          scale_for_simulation=scale_simulation,
                                          s_grid=50,
                                          scale_for_pseudotime=scale_dev,
                                          vm=vm)
```

Example 119 (unknown):
```unknown
dev.inner_product_df
```

Example 120 (unknown):
```unknown
df = dev.inner_product_df.copy()
df[['score','pseudotime']].corr(method='pearson')
```

Example 121 (unknown):
```unknown
dev.inner_product_df
```

Example 122 (unknown):
```unknown
len(dev.inner_product)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/2_multivi_compute_latent_space.html

**Contents:**
- Set Up¶
- Read in¶
- Organise anndata¶
- Setup and training MultiVI (with continuous covariates)¶

5664 rows × 40 columns

544641 rows × 3 columns

MultiVI requires the main batch annotation to correspond to the modality of the samples. Other batch annotation, such as in the case of multiple RNA-only batches, can be specified using categorical_covariate_keys.

**Examples:**

Example 1 (python):
```python
job_name = None
n_layers = None
developing=None
paired_adata_path=None
unimodality_adata_path=None
multiome_only=None
combination=None
```

Example 2 (python):
```python
# Parameters
job_name = "aCMs_WholeSample"
n_layers = 1
developing = False
paired_adata_path = "/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA/aCMs_WholeSample_paired.h5ad"
unimodality_adata_path = "none"
combination = "multivi_aCMs_WholeSample_n_layers1"
```

Example 3 (python):
```python
import scvi
import anndata
import numpy as np
import pandas as pd
import scanpy as sc
import matplotlib.pyplot as plt
import os
```

Example 4 (unknown):
```unknown
Global seed set to 0
```

Example 5 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
```

Example 6 (python):
```python
import scanpy as sc

def all_integers_in_float_list(float_list):
    """Check if all values in a list of floats are integers."""
    return all(float_val.is_integer() for float_val in float_list)

def check_counts_layer(adata):
    """
    Check if the "counts" layer exists in the given AnnData object.
    
    Parameters:
    - adata (anndata.AnnData): The AnnData object to check.
    
    Returns:
    - bool: True if "counts" layer exists, False otherwise.
    """
    return "counts" in adata.layers.keys()

def makecountslayer(adata, apply_norm_log=True):
    """
    Ensure that the AnnData object has a "counts" layer and optionally log-normalize .X
    
    Parameters:
    - adata (anndata.AnnData): The AnnData object to be processed.
    - apply_norm_log (bool): Whether to apply normalization and log1p transformation. Default is True.
    
    Returns:
    - None: Modifies the AnnData object in place.
    """
    # Check if 'counts' layer exists
    if check_counts_layer(adata):
        print("Counts layer found, copying counts to .X...")
        # Copy data from 'counts' layer to .X
        adata.X = adata.layers["counts"].copy()
    else:
        print("No counts layer found, checking whether .X contains count data (integers)...")
        # Sample first 1000 elements from .X for integer check
        adata_X = adata.X.data[:1000].tolist()

        # Check if sampled data are all integers
        if all_integers_in_float_list(adata_X):
            print("Counts found in adata.X, saving them to a counts layer...")
            # Copy data from .X to 'counts' layer
            adata.layers["counts"] = adata.X.copy()
        else:
            raise ValueError("No count data found, cannot proceed")
    
    if apply_norm_log:
        print("Norm-logging .X...")
        # Normalize and log-transform data
        sc.pp.normalize_total(adata, target_sum=1e4)
        sc.pp.log1p(adata)
    else:
        print("NOT norm-logging")
```

Example 7 (python):
```python
from collections import Counter

def ensure_unique_var_names(adata):
    """
    Ensure that the variable names in an AnnData object are unique.
    
    Parameters:
    -----------
    adata : AnnData
        The AnnData object to be checked and processed.

    Returns:
    --------
    None
        Modifies the AnnData object in place if non-unique variable names are found.
    """
    
    # Find non-unique variable names
    var_names = list(adata.var_names)
    counter = Counter(var_names)
    non_unique_var_names = [name for name, count in counter.items() if count > 1]

    # If non-unique variable names are found, make them unique
    if non_unique_var_names:
        print("The following variable names are NOT unique:", non_unique_var_names)
        print("Running adata.var_names_make_unique()...")
        adata.var_names_make_unique()
    else:
        print("All variable names are unique.")
```

Example 8 (python):
```python
if unimodality_adata_path=="none":
    multiome_only=True
else:
    multiome_only=False
```

Example 9 (unknown):
```unknown
continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo','S_score','G2M_score']
```

Example 10 (unknown):
```unknown
model_dir=os.path.join("/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA","models") # for model
latent_variables_dir=os.path.join("/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA","latent_variables") # for latent variables
```

Example 11 (unknown):
```unknown
if job_name=="global":
    max_epochs=200
else:
    max_epochs=500
if developing:
    max_epochs=2
```

Example 12 (python):
```python
import session_info
session_info.show()
```

Example 13 (python):
```python
-----
anndata             0.8.0
matplotlib          3.7.2
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
scvi                0.20.3
session_info        1.0.0
-----
```

Example 14 (unknown):
```unknown
PIL                         10.0.0
absl                        NA
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.07.22
cffi                        1.15.1
charset_normalizer          3.2.0
chex                        0.1.7
comm                        0.1.3
contextlib2                 NA
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.7
decorator                   5.1.1
defusedxml                  0.7.1
docrep                      0.3.2
etils                       1.4.0
executing                   1.2.0
fastjsonschema              NA
flax                        0.7.0
fqdn                        NA
fsspec                      2023.6.0
graphviz                    0.20.1
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_resources         NA
ipykernel                   6.25.0
isoduration                 NA
jax                         0.4.14
jaxlib                      0.4.14
jedi                        0.18.2
jinja2                      3.1.2
joblib                      1.3.1
json5                       NA
jsonpointer                 2.0
jsonschema                  4.18.4
jsonschema_specifications   NA
jupyter_events              0.6.3
jupyter_server              2.7.0
jupyterlab_server           2.24.0
kiwisolver                  1.4.4
leidenalg                   0.10.1
lightning_fabric            1.9.5
lightning_utilities         0.9.0
llvmlite                    0.40.1
markupsafe                  2.1.3
ml_collections              NA
ml_dtypes                   0.2.0
mpl_toolkits                NA
mpmath                      1.3.0
msgpack                     1.0.5
mudata                      0.2.3
multipledispatch            0.6.0
natsort                     8.4.0
nbformat                    5.9.1
numba                       0.57.1
numpyro                     0.12.1
nvfuser                     NA
opt_einsum                  v3.3.0
optax                       0.1.7
overrides                   NA
packaging                   23.1
parso                       0.8.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
platformdirs                3.9.1
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pyarrow                     12.0.1
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.15.1
pyparsing                   3.0.9
pyro                        1.8.6
pythonjsonlogger            NA
pytorch_lightning           1.9.5
pytz                        2023.3
referencing                 NA
regex                       2.5.129
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rich                        NA
rpds                        NA
scipy                       1.11.1
send2trash                  NA
setuptools                  68.0.0
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
stack_data                  0.6.2
sympy                       1.12
texttable                   1.6.7
threadpoolctl               3.2.0
toolz                       0.12.0
torch                       2.0.1+cu117
torchmetrics                1.0.1
tornado                     6.3.2
tqdm                        4.65.0
traitlets                   5.9.0
tree                        0.1.8
typing_extensions           NA
uri_template                NA
urllib3                     1.26.16
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.1
yaml                        6.0
zmq                         25.1.0
```

Example 15 (unknown):
```unknown
-----
IPython             8.14.0
jupyter_client      8.3.0
jupyter_core        5.3.1
jupyterlab          4.0.3
notebook            7.0.0
-----
Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0]
Linux-5.4.0-159-generic-x86_64-with-glibc2.27
-----
Session information updated at 2023-10-19 16:46
```

Example 16 (python):
```python
sc.settings.verbosity = 3             # verbosity: errors (0), warnings (1), info (2), hints (3)
sc.settings.set_figure_params(dpi=50, facecolor='white', color_map = 'RdPu',figsize=(4, 4))
sc.set_figure_params(dpi=80, dpi_save=150, figsize=(6,6))
```

Example 17 (unknown):
```unknown
!nvidia-smi
```

Example 18 (unknown):
```unknown
Thu Oct 19 16:46:43 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03   Driver Version: 510.108.03   CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
```

Example 19 (unknown):
```unknown
|   0  NVIDIA A100-SXM...  On   | 00000000:07:00.0 Off |                    0 |
| N/A   44C    P0   104W / 400W |  21381MiB / 81920MiB |     75%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
```

Example 20 (unknown):
```unknown
|    0   N/A  N/A    247245      C   python                          21379MiB |
+-----------------------------------------------------------------------------+
```

Example 21 (python):
```python
adata_paired = sc.read_h5ad(paired_adata_path)
print(adata_paired.shape)

if developing:
    adata_paired=adata_paired[::10, :] # slices to keep every Xth cell
    
print(adata_paired.shape)
adata_paired
```

Example 22 (unknown):
```unknown
(5664, 544641)
(5664, 544641)
```

Example 23 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
```

Example 24 (python):
```python
adata_paired.obs
```

Example 25 (python):
```python
adata_paired.var
```

Example 26 (python):
```python
sum(adata_paired.var['modality']=='Gene Expression')
```

Example 27 (python):
```python
ensure_unique_var_names(adata_paired)
```

Example 28 (unknown):
```unknown
All variable names are unique.
```

Example 29 (python):
```python
makecountslayer(adata_paired, apply_norm_log=False)
```

Example 30 (python):
```python
No counts layer found, checking whether .X contains count data (integers)...
Counts found in adata.X, saving them to a counts layer...
NOT norm-logging
```

Example 31 (python):
```python
if multiome_only:
    adata_mvi=scvi.data.organize_multiome_anndatas(multi_anndata=adata_paired)
    adata_mvi = adata_mvi[:, adata_mvi.var["modality"].argsort()].copy()
if multiome_only==False:
    unimodality_adata = sc.read_h5ad(unimodality_adata_path)
    print(unimodality_adata.shape)
    if developing:
        unimodality_adata=unimodality_adata[::10, :] # slices to keep every Xth cell
    ensure_unique_var_names(unimodality_adata)
    makecountslayer(unimodality_adata, apply_norm_log=False)
    adata_mvi=scvi.data.organize_multiome_anndatas(multi_anndata=adata_paired,rna_anndata=unimodality_adata)
    adata_mvi=adata_mvi[:, adata_mvi.var["modality"].argsort()].copy()
```

Example 32 (python):
```python
# We also filter features to remove those that appear in fewer than 1% of the cells
print(adata_mvi.shape)
sc.pp.filter_genes(adata_mvi, min_cells=int(adata_mvi.shape[0] * 0.01))
print(adata_mvi.shape)
```

Example 33 (unknown):
```unknown
(5664, 544641)
```

Example 34 (unknown):
```unknown
filtered out 364821 genes that are detected in less than 56 cells
```

Example 35 (unknown):
```unknown
(5664, 179820)
```

Example 36 (python):
```python
adata_mvi.obs['modality'].value_counts()
```

Example 37 (unknown):
```unknown
paired    5664
Name: modality, dtype: int64
```

Example 38 (python):
```python
adata_mvi.obs['batch_key'].value_counts()
```

Example 39 (unknown):
```unknown
Hst33_Nuclei_Multiome-v1    3925
C92_Nuclei_Multiome-v1       443
C85_Nuclei_Multiome-v1       380
Hst36_Nuclei_Multiome-v1     372
C87_Nuclei_Multiome-v1       321
C98_Nuclei_Multiome-v1       186
C82_Nuclei_Multiome-v1        37
Name: batch_key, dtype: int64
```

Example 40 (python):
```python
scvi.model.MULTIVI.setup_anndata(adata_mvi, batch_key='modality',
                                categorical_covariate_keys=['batch_key'],
                                continuous_covariate_keys=continuous_covariate_keys)
```

Example 41 (unknown):
```unknown
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
```

Example 42 (python):
```python
# When creating the object, we need to specify how many of the features are genes, and how many are genomic regions. 
# This is so MultiVI can determine the exact architecture for each modality.
mvi = scvi.model.MULTIVI(
    adata_mvi,
    n_genes=(adata_mvi.var['modality']=='Gene Expression').sum(),
    n_regions=(adata_mvi.var['modality']=='Peaks').sum(),
    n_layers_encoder=n_layers,
    n_layers_decoder=n_layers,
)
mvi.view_anndata_setup()
```

Example 43 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
  warnings.warn("Initializing zero-element tensors is a no-op")
```

Example 44 (unknown):
```unknown
Anndata setup with scvi-tools version 0.20.3.
```

Example 45 (unknown):
```unknown
Setup via `MULTIVI.setup_anndata` with arguments:
```

Example 46 (unknown):
```unknown
{
│   'layer': None,
│   'batch_key': 'modality',
│   'size_factor_key': None,
│   'categorical_covariate_keys': ['batch_key'],
│   'continuous_covariate_keys': ['total_counts', 'pct_counts_mt', 'pct_counts_ribo', 'S_score', 'G2M_score'],
│   'protein_expression_obsm_key': None,
│   'protein_names_uns_key': None
}
```

Example 47 (unknown):
```unknown
Summary Statistics          
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃     Summary Stat Key     ┃ Value  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│         n_batch          │   1    │
│         n_cells          │  5664  │
│ n_extra_categorical_covs │   1    │
│ n_extra_continuous_covs  │   5    │
│         n_labels         │   1    │
│          n_vars          │ 179820 │
└──────────────────────────┴────────┘
```

Example 48 (python):
```python
Data Registry                             
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃      Registry Key      ┃            scvi-tools Location             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│           X            │                  adata.X                   │
│         batch          │          adata.obs['_scvi_batch']          │
│ extra_categorical_covs │ adata.obsm['_scvi_extra_categorical_covs'] │
│ extra_continuous_covs  │ adata.obsm['_scvi_extra_continuous_covs']  │
│         ind_x          │           adata.obs['_indices']            │
│         labels         │         adata.obs['_scvi_labels']          │
└────────────────────────┴────────────────────────────────────────────┘
```

Example 49 (python):
```python
batch State Registry                    
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃    Source Location    ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['modality'] │   paired   │          0          │
└───────────────────────┴────────────┴─────────────────────┘
```

Example 50 (python):
```python
labels State Registry                      
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃      Source Location      ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['_scvi_labels'] │     0      │          0          │
└───────────────────────────┴────────────┴─────────────────────┘
```

Example 51 (python):
```python
batch State Registry                    
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃    Source Location    ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['modality'] │   paired   │          0          │
└───────────────────────┴────────────┴─────────────────────┘
```

Example 52 (python):
```python
extra_categorical_covs State Registry                   
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃    Source Location     ┃        Categories        ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['batch_key'] │  C82_Nuclei_Multiome-v1  │          0          │
│                        │  C85_Nuclei_Multiome-v1  │          1          │
│                        │  C87_Nuclei_Multiome-v1  │          2          │
│                        │  C92_Nuclei_Multiome-v1  │          3          │
│                        │  C98_Nuclei_Multiome-v1  │          4          │
│                        │ Hst33_Nuclei_Multiome-v1 │          5          │
│                        │ Hst36_Nuclei_Multiome-v1 │          6          │
│                        │                          │                     │
└────────────────────────┴──────────────────────────┴─────────────────────┘
```

Example 53 (python):
```python
extra_continuous_covs State   
            Registry            
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃       Source Location        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│  adata.obs['total_counts']   │
│  adata.obs['pct_counts_mt']  │
│ adata.obs['pct_counts_ribo'] │
│     adata.obs['S_score']     │
│    adata.obs['G2M_score']    │
└──────────────────────────────┘
```

Example 54 (unknown):
```unknown
mvi.train(max_epochs=max_epochs)
```

Example 55 (unknown):
```unknown
GPU available: True (cuda), used: True
```

Example 56 (unknown):
```unknown
TPU available: False, using: 0 TPU cores
```

Example 57 (unknown):
```unknown
IPU available: False, using: 0 IPUs
```

Example 58 (unknown):
```unknown
HPU available: False, using: 0 HPUs
```

Example 59 (unknown):
```unknown
You are using a CUDA device ('NVIDIA A100-SXM4-80GB') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
```

Example 60 (unknown):
```unknown
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
```

Example 61 (unknown):
```unknown
Training:   0%|          | 0/500 [00:00<?, ?it/s]
```

Example 62 (unknown):
```unknown
Epoch 1/500:   0%|          | 0/500 [00:00<?, ?it/s]
```

Example 63 (unknown):
```unknown
Epoch 1/500:   0%|          | 1/500 [00:17<2:22:02, 17.08s/it]
```

Example 64 (unknown):
```unknown
Epoch 1/500:   0%|          | 1/500 [00:17<2:22:02, 17.08s/it, loss=2.18e+04, v_num=1]
```

Example 65 (unknown):
```unknown
Epoch 2/500:   0%|          | 1/500 [00:17<2:22:02, 17.08s/it, loss=2.18e+04, v_num=1]
```

Example 66 (unknown):
```unknown
Epoch 2/500:   0%|          | 2/500 [00:24<1:32:11, 11.11s/it, loss=2.18e+04, v_num=1]
```

Example 67 (unknown):
```unknown
Epoch 2/500:   0%|          | 2/500 [00:24<1:32:11, 11.11s/it, loss=1.87e+04, v_num=1]
```

Example 68 (unknown):
```unknown
Epoch 3/500:   0%|          | 2/500 [00:24<1:32:11, 11.11s/it, loss=1.87e+04, v_num=1]
```

Example 69 (unknown):
```unknown
Epoch 3/500:   1%|          | 3/500 [00:31<1:16:30,  9.24s/it, loss=1.87e+04, v_num=1]
```

Example 70 (unknown):
```unknown
Epoch 3/500:   1%|          | 3/500 [00:31<1:16:30,  9.24s/it, loss=1.7e+04, v_num=1]
```

Example 71 (unknown):
```unknown
Epoch 4/500:   1%|          | 3/500 [00:31<1:16:30,  9.24s/it, loss=1.7e+04, v_num=1]
```

Example 72 (unknown):
```unknown
Epoch 4/500:   1%|          | 4/500 [00:38<1:08:59,  8.35s/it, loss=1.7e+04, v_num=1]
```

Example 73 (unknown):
```unknown
Epoch 4/500:   1%|          | 4/500 [00:38<1:08:59,  8.35s/it, loss=1.65e+04, v_num=1]
```

Example 74 (unknown):
```unknown
Epoch 5/500:   1%|          | 4/500 [00:38<1:08:59,  8.35s/it, loss=1.65e+04, v_num=1]
```

Example 75 (unknown):
```unknown
Epoch 5/500:   1%|          | 5/500 [00:44<1:04:07,  7.77s/it, loss=1.65e+04, v_num=1]
```

Example 76 (unknown):
```unknown
Epoch 5/500:   1%|          | 5/500 [00:44<1:04:07,  7.77s/it, loss=1.62e+04, v_num=1]
```

Example 77 (unknown):
```unknown
Epoch 6/500:   1%|          | 5/500 [00:44<1:04:07,  7.77s/it, loss=1.62e+04, v_num=1]
```

Example 78 (unknown):
```unknown
Epoch 6/500:   1%|          | 6/500 [00:51<1:00:48,  7.39s/it, loss=1.62e+04, v_num=1]
```

Example 79 (unknown):
```unknown
Epoch 6/500:   1%|          | 6/500 [00:51<1:00:48,  7.39s/it, loss=1.58e+04, v_num=1]
```

Example 80 (unknown):
```unknown
Epoch 7/500:   1%|          | 6/500 [00:51<1:00:48,  7.39s/it, loss=1.58e+04, v_num=1]
```

Example 81 (unknown):
```unknown
Epoch 7/500:   1%|▏         | 7/500 [00:58<59:08,  7.20s/it, loss=1.58e+04, v_num=1]
```

Example 82 (unknown):
```unknown
Epoch 7/500:   1%|▏         | 7/500 [00:58<59:08,  7.20s/it, loss=1.59e+04, v_num=1]
```

Example 83 (unknown):
```unknown
Epoch 8/500:   1%|▏         | 7/500 [00:58<59:08,  7.20s/it, loss=1.59e+04, v_num=1]
```

Example 84 (unknown):
```unknown
Epoch 8/500:   2%|▏         | 8/500 [01:05<58:21,  7.12s/it, loss=1.59e+04, v_num=1]
```

Example 85 (unknown):
```unknown
Epoch 8/500:   2%|▏         | 8/500 [01:05<58:21,  7.12s/it, loss=1.58e+04, v_num=1]
```

Example 86 (unknown):
```unknown
Epoch 9/500:   2%|▏         | 8/500 [01:05<58:21,  7.12s/it, loss=1.58e+04, v_num=1]
```

Example 87 (unknown):
```unknown
Epoch 9/500:   2%|▏         | 9/500 [01:11<57:03,  6.97s/it, loss=1.58e+04, v_num=1]
```

Example 88 (unknown):
```unknown
Epoch 9/500:   2%|▏         | 9/500 [01:11<57:03,  6.97s/it, loss=1.55e+04, v_num=1]
```

Example 89 (unknown):
```unknown
Epoch 10/500:   2%|▏         | 9/500 [01:11<57:03,  6.97s/it, loss=1.55e+04, v_num=1]
```

Example 90 (unknown):
```unknown
Epoch 10/500:   2%|▏         | 10/500 [01:18<56:49,  6.96s/it, loss=1.55e+04, v_num=1]
```

Example 91 (unknown):
```unknown
Epoch 10/500:   2%|▏         | 10/500 [01:18<56:49,  6.96s/it, loss=1.55e+04, v_num=1]
```

Example 92 (unknown):
```unknown
Epoch 11/500:   2%|▏         | 10/500 [01:18<56:49,  6.96s/it, loss=1.55e+04, v_num=1]
```

Example 93 (unknown):
```unknown
Epoch 11/500:   2%|▏         | 11/500 [01:25<56:10,  6.89s/it, loss=1.55e+04, v_num=1]
```

Example 94 (unknown):
```unknown
Epoch 11/500:   2%|▏         | 11/500 [01:25<56:10,  6.89s/it, loss=1.53e+04, v_num=1]
```

Example 95 (unknown):
```unknown
Epoch 12/500:   2%|▏         | 11/500 [01:25<56:10,  6.89s/it, loss=1.53e+04, v_num=1]
```

Example 96 (unknown):
```unknown
Epoch 12/500:   2%|▏         | 12/500 [01:32<56:16,  6.92s/it, loss=1.53e+04, v_num=1]
```

Example 97 (unknown):
```unknown
Epoch 12/500:   2%|▏         | 12/500 [01:32<56:16,  6.92s/it, loss=1.57e+04, v_num=1]
```

Example 98 (unknown):
```unknown
Epoch 13/500:   2%|▏         | 12/500 [01:32<56:16,  6.92s/it, loss=1.57e+04, v_num=1]
```

Example 99 (unknown):
```unknown
Epoch 13/500:   3%|▎         | 13/500 [01:39<56:19,  6.94s/it, loss=1.57e+04, v_num=1]
```

Example 100 (unknown):
```unknown
Epoch 13/500:   3%|▎         | 13/500 [01:39<56:19,  6.94s/it, loss=1.55e+04, v_num=1]
```

Example 101 (unknown):
```unknown
Epoch 14/500:   3%|▎         | 13/500 [01:39<56:19,  6.94s/it, loss=1.55e+04, v_num=1]
```

Example 102 (unknown):
```unknown
Epoch 14/500:   3%|▎         | 14/500 [01:46<56:15,  6.95s/it, loss=1.55e+04, v_num=1]
```

Example 103 (unknown):
```unknown
Epoch 14/500:   3%|▎         | 14/500 [01:46<56:15,  6.95s/it, loss=1.53e+04, v_num=1]
```

Example 104 (unknown):
```unknown
Epoch 15/500:   3%|▎         | 14/500 [01:46<56:15,  6.95s/it, loss=1.53e+04, v_num=1]
```

Example 105 (unknown):
```unknown
Epoch 15/500:   3%|▎         | 15/500 [01:53<56:12,  6.95s/it, loss=1.53e+04, v_num=1]
```

Example 106 (unknown):
```unknown
Epoch 15/500:   3%|▎         | 15/500 [01:53<56:12,  6.95s/it, loss=1.55e+04, v_num=1]
```

Example 107 (unknown):
```unknown
Epoch 16/500:   3%|▎         | 15/500 [01:53<56:12,  6.95s/it, loss=1.55e+04, v_num=1]
```

Example 108 (unknown):
```unknown
Epoch 16/500:   3%|▎         | 16/500 [02:00<56:05,  6.95s/it, loss=1.55e+04, v_num=1]
```

Example 109 (unknown):
```unknown
Epoch 16/500:   3%|▎         | 16/500 [02:00<56:05,  6.95s/it, loss=1.53e+04, v_num=1]
```

Example 110 (unknown):
```unknown
Epoch 17/500:   3%|▎         | 16/500 [02:00<56:05,  6.95s/it, loss=1.53e+04, v_num=1]
```

Example 111 (unknown):
```unknown
Epoch 17/500:   3%|▎         | 17/500 [02:07<55:57,  6.95s/it, loss=1.53e+04, v_num=1]
```

Example 112 (unknown):
```unknown
Epoch 17/500:   3%|▎         | 17/500 [02:07<55:57,  6.95s/it, loss=1.53e+04, v_num=1]
```

Example 113 (unknown):
```unknown
Epoch 18/500:   3%|▎         | 17/500 [02:07<55:57,  6.95s/it, loss=1.53e+04, v_num=1]
```

Example 114 (unknown):
```unknown
Epoch 18/500:   4%|▎         | 18/500 [02:14<55:49,  6.95s/it, loss=1.53e+04, v_num=1]
```

Example 115 (unknown):
```unknown
Epoch 18/500:   4%|▎         | 18/500 [02:14<55:49,  6.95s/it, loss=1.55e+04, v_num=1]
```

Example 116 (unknown):
```unknown
Epoch 19/500:   4%|▎         | 18/500 [02:14<55:49,  6.95s/it, loss=1.55e+04, v_num=1]
```

Example 117 (unknown):
```unknown
Epoch 19/500:   4%|▍         | 19/500 [02:20<55:17,  6.90s/it, loss=1.55e+04, v_num=1]
```

Example 118 (unknown):
```unknown
Epoch 19/500:   4%|▍         | 19/500 [02:20<55:17,  6.90s/it, loss=1.53e+04, v_num=1]
```

Example 119 (unknown):
```unknown
Epoch 20/500:   4%|▍         | 19/500 [02:20<55:17,  6.90s/it, loss=1.53e+04, v_num=1]
```

Example 120 (unknown):
```unknown
Epoch 20/500:   4%|▍         | 20/500 [02:27<55:16,  6.91s/it, loss=1.53e+04, v_num=1]
```

Example 121 (unknown):
```unknown
Epoch 20/500:   4%|▍         | 20/500 [02:27<55:16,  6.91s/it, loss=1.55e+04, v_num=1]
```

Example 122 (unknown):
```unknown
Epoch 21/500:   4%|▍         | 20/500 [02:27<55:16,  6.91s/it, loss=1.55e+04, v_num=1]
```

Example 123 (unknown):
```unknown
Epoch 21/500:   4%|▍         | 21/500 [02:34<54:41,  6.85s/it, loss=1.55e+04, v_num=1]
```

Example 124 (unknown):
```unknown
Epoch 21/500:   4%|▍         | 21/500 [02:34<54:41,  6.85s/it, loss=1.55e+04, v_num=1]
```

Example 125 (unknown):
```unknown
Epoch 22/500:   4%|▍         | 21/500 [02:34<54:41,  6.85s/it, loss=1.55e+04, v_num=1]
```

Example 126 (unknown):
```unknown
Epoch 22/500:   4%|▍         | 22/500 [02:41<54:20,  6.82s/it, loss=1.55e+04, v_num=1]
```

Example 127 (unknown):
```unknown
Epoch 22/500:   4%|▍         | 22/500 [02:41<54:20,  6.82s/it, loss=1.52e+04, v_num=1]
```

Example 128 (unknown):
```unknown
Epoch 23/500:   4%|▍         | 22/500 [02:41<54:20,  6.82s/it, loss=1.52e+04, v_num=1]
```

Example 129 (unknown):
```unknown
Epoch 23/500:   5%|▍         | 23/500 [02:48<54:26,  6.85s/it, loss=1.52e+04, v_num=1]
```

Example 130 (unknown):
```unknown
Epoch 23/500:   5%|▍         | 23/500 [02:48<54:26,  6.85s/it, loss=1.52e+04, v_num=1]
```

Example 131 (unknown):
```unknown
Epoch 24/500:   5%|▍         | 23/500 [02:48<54:26,  6.85s/it, loss=1.52e+04, v_num=1]
```

Example 132 (unknown):
```unknown
Epoch 24/500:   5%|▍         | 24/500 [02:54<53:41,  6.77s/it, loss=1.52e+04, v_num=1]
```

Example 133 (unknown):
```unknown
Epoch 24/500:   5%|▍         | 24/500 [02:54<53:41,  6.77s/it, loss=1.52e+04, v_num=1]
```

Example 134 (unknown):
```unknown
Epoch 25/500:   5%|▍         | 24/500 [02:54<53:41,  6.77s/it, loss=1.52e+04, v_num=1]
```

Example 135 (unknown):
```unknown
Epoch 25/500:   5%|▌         | 25/500 [03:01<53:52,  6.80s/it, loss=1.52e+04, v_num=1]
```

Example 136 (unknown):
```unknown
Epoch 25/500:   5%|▌         | 25/500 [03:01<53:52,  6.80s/it, loss=1.53e+04, v_num=1]
```

Example 137 (unknown):
```unknown
Epoch 26/500:   5%|▌         | 25/500 [03:01<53:52,  6.80s/it, loss=1.53e+04, v_num=1]
```

Example 138 (unknown):
```unknown
Epoch 26/500:   5%|▌         | 26/500 [03:08<53:36,  6.79s/it, loss=1.53e+04, v_num=1]
```

Example 139 (unknown):
```unknown
Epoch 26/500:   5%|▌         | 26/500 [03:08<53:36,  6.79s/it, loss=1.53e+04, v_num=1]
```

Example 140 (unknown):
```unknown
Epoch 27/500:   5%|▌         | 26/500 [03:08<53:36,  6.79s/it, loss=1.53e+04, v_num=1]
```

Example 141 (unknown):
```unknown
Epoch 27/500:   5%|▌         | 27/500 [03:15<53:54,  6.84s/it, loss=1.53e+04, v_num=1]
```

Example 142 (unknown):
```unknown
Epoch 27/500:   5%|▌         | 27/500 [03:15<53:54,  6.84s/it, loss=1.57e+04, v_num=1]
```

Example 143 (unknown):
```unknown
Epoch 28/500:   5%|▌         | 27/500 [03:15<53:54,  6.84s/it, loss=1.57e+04, v_num=1]
```

Example 144 (unknown):
```unknown
Epoch 28/500:   6%|▌         | 28/500 [03:22<54:03,  6.87s/it, loss=1.57e+04, v_num=1]
```

Example 145 (unknown):
```unknown
Epoch 28/500:   6%|▌         | 28/500 [03:22<54:03,  6.87s/it, loss=1.57e+04, v_num=1]
```

Example 146 (unknown):
```unknown
Epoch 29/500:   6%|▌         | 28/500 [03:22<54:03,  6.87s/it, loss=1.57e+04, v_num=1]
```

Example 147 (unknown):
```unknown
Epoch 29/500:   6%|▌         | 29/500 [03:29<54:12,  6.91s/it, loss=1.57e+04, v_num=1]
```

Example 148 (unknown):
```unknown
Epoch 29/500:   6%|▌         | 29/500 [03:29<54:12,  6.91s/it, loss=1.53e+04, v_num=1]
```

Example 149 (unknown):
```unknown
Epoch 30/500:   6%|▌         | 29/500 [03:29<54:12,  6.91s/it, loss=1.53e+04, v_num=1]
```

Example 150 (unknown):
```unknown
Epoch 30/500:   6%|▌         | 30/500 [03:36<54:11,  6.92s/it, loss=1.53e+04, v_num=1]
```

Example 151 (unknown):
```unknown
Epoch 30/500:   6%|▌         | 30/500 [03:36<54:11,  6.92s/it, loss=1.5e+04, v_num=1]
```

Example 152 (unknown):
```unknown
Epoch 31/500:   6%|▌         | 30/500 [03:36<54:11,  6.92s/it, loss=1.5e+04, v_num=1]
```

Example 153 (unknown):
```unknown
Epoch 31/500:   6%|▌         | 31/500 [03:43<53:31,  6.85s/it, loss=1.5e+04, v_num=1]
```

Example 154 (unknown):
```unknown
Epoch 31/500:   6%|▌         | 31/500 [03:43<53:31,  6.85s/it, loss=1.52e+04, v_num=1]
```

Example 155 (unknown):
```unknown
Epoch 32/500:   6%|▌         | 31/500 [03:43<53:31,  6.85s/it, loss=1.52e+04, v_num=1]
```

Example 156 (unknown):
```unknown
Epoch 32/500:   6%|▋         | 32/500 [03:49<53:22,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 157 (unknown):
```unknown
Epoch 32/500:   6%|▋         | 32/500 [03:49<53:22,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 158 (unknown):
```unknown
Epoch 33/500:   6%|▋         | 32/500 [03:49<53:22,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 159 (unknown):
```unknown
Epoch 33/500:   7%|▋         | 33/500 [03:56<53:03,  6.82s/it, loss=1.51e+04, v_num=1]
```

Example 160 (unknown):
```unknown
Epoch 33/500:   7%|▋         | 33/500 [03:56<53:03,  6.82s/it, loss=1.56e+04, v_num=1]
```

Example 161 (unknown):
```unknown
Epoch 34/500:   7%|▋         | 33/500 [03:56<53:03,  6.82s/it, loss=1.56e+04, v_num=1]
```

Example 162 (unknown):
```unknown
Epoch 34/500:   7%|▋         | 34/500 [04:03<52:46,  6.79s/it, loss=1.56e+04, v_num=1]
```

Example 163 (unknown):
```unknown
Epoch 34/500:   7%|▋         | 34/500 [04:03<52:46,  6.79s/it, loss=1.58e+04, v_num=1]
```

Example 164 (unknown):
```unknown
Epoch 35/500:   7%|▋         | 34/500 [04:03<52:46,  6.79s/it, loss=1.58e+04, v_num=1]
```

Example 165 (unknown):
```unknown
Epoch 35/500:   7%|▋         | 35/500 [04:10<52:30,  6.78s/it, loss=1.58e+04, v_num=1]
```

Example 166 (unknown):
```unknown
Epoch 35/500:   7%|▋         | 35/500 [04:10<52:30,  6.78s/it, loss=1.57e+04, v_num=1]
```

Example 167 (unknown):
```unknown
Epoch 36/500:   7%|▋         | 35/500 [04:10<52:30,  6.78s/it, loss=1.57e+04, v_num=1]
```

Example 168 (unknown):
```unknown
Epoch 36/500:   7%|▋         | 36/500 [04:17<52:50,  6.83s/it, loss=1.57e+04, v_num=1]
```

Example 169 (unknown):
```unknown
Epoch 36/500:   7%|▋         | 36/500 [04:17<52:50,  6.83s/it, loss=1.54e+04, v_num=1]
```

Example 170 (unknown):
```unknown
Epoch 37/500:   7%|▋         | 36/500 [04:17<52:50,  6.83s/it, loss=1.54e+04, v_num=1]
```

Example 171 (unknown):
```unknown
Epoch 37/500:   7%|▋         | 37/500 [04:23<52:20,  6.78s/it, loss=1.54e+04, v_num=1]
```

Example 172 (unknown):
```unknown
Epoch 37/500:   7%|▋         | 37/500 [04:23<52:20,  6.78s/it, loss=1.52e+04, v_num=1]
```

Example 173 (unknown):
```unknown
Epoch 38/500:   7%|▋         | 37/500 [04:23<52:20,  6.78s/it, loss=1.52e+04, v_num=1]
```

Example 174 (unknown):
```unknown
Epoch 38/500:   8%|▊         | 38/500 [04:30<51:51,  6.74s/it, loss=1.52e+04, v_num=1]
```

Example 175 (unknown):
```unknown
Epoch 38/500:   8%|▊         | 38/500 [04:30<51:51,  6.74s/it, loss=1.53e+04, v_num=1]
```

Example 176 (unknown):
```unknown
Epoch 39/500:   8%|▊         | 38/500 [04:30<51:51,  6.74s/it, loss=1.53e+04, v_num=1]
```

Example 177 (unknown):
```unknown
Epoch 39/500:   8%|▊         | 39/500 [04:37<51:50,  6.75s/it, loss=1.53e+04, v_num=1]
```

Example 178 (unknown):
```unknown
Epoch 39/500:   8%|▊         | 39/500 [04:37<51:50,  6.75s/it, loss=1.55e+04, v_num=1]
```

Example 179 (unknown):
```unknown
Epoch 40/500:   8%|▊         | 39/500 [04:37<51:50,  6.75s/it, loss=1.55e+04, v_num=1]
```

Example 180 (unknown):
```unknown
Epoch 40/500:   8%|▊         | 40/500 [04:43<51:42,  6.74s/it, loss=1.55e+04, v_num=1]
```

Example 181 (unknown):
```unknown
Epoch 40/500:   8%|▊         | 40/500 [04:43<51:42,  6.74s/it, loss=1.52e+04, v_num=1]
```

Example 182 (unknown):
```unknown
Epoch 41/500:   8%|▊         | 40/500 [04:43<51:42,  6.74s/it, loss=1.52e+04, v_num=1]
```

Example 183 (unknown):
```unknown
Epoch 41/500:   8%|▊         | 41/500 [04:50<52:01,  6.80s/it, loss=1.52e+04, v_num=1]
```

Example 184 (unknown):
```unknown
Epoch 41/500:   8%|▊         | 41/500 [04:50<52:01,  6.80s/it, loss=1.54e+04, v_num=1]
```

Example 185 (unknown):
```unknown
Epoch 42/500:   8%|▊         | 41/500 [04:50<52:01,  6.80s/it, loss=1.54e+04, v_num=1]
```

Example 186 (unknown):
```unknown
Epoch 42/500:   8%|▊         | 42/500 [04:57<52:18,  6.85s/it, loss=1.54e+04, v_num=1]
```

Example 187 (unknown):
```unknown
Epoch 42/500:   8%|▊         | 42/500 [04:57<52:18,  6.85s/it, loss=1.53e+04, v_num=1]
```

Example 188 (unknown):
```unknown
Epoch 43/500:   8%|▊         | 42/500 [04:57<52:18,  6.85s/it, loss=1.53e+04, v_num=1]
```

Example 189 (unknown):
```unknown
Epoch 43/500:   9%|▊         | 43/500 [05:04<52:36,  6.91s/it, loss=1.53e+04, v_num=1]
```

Example 190 (unknown):
```unknown
Epoch 43/500:   9%|▊         | 43/500 [05:04<52:36,  6.91s/it, loss=1.51e+04, v_num=1]
```

Example 191 (unknown):
```unknown
Epoch 44/500:   9%|▊         | 43/500 [05:04<52:36,  6.91s/it, loss=1.51e+04, v_num=1]
```

Example 192 (unknown):
```unknown
Epoch 44/500:   9%|▉         | 44/500 [05:11<52:33,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 193 (unknown):
```unknown
Epoch 44/500:   9%|▉         | 44/500 [05:11<52:33,  6.92s/it, loss=1.52e+04, v_num=1]
```

Example 194 (unknown):
```unknown
Epoch 45/500:   9%|▉         | 44/500 [05:11<52:33,  6.92s/it, loss=1.52e+04, v_num=1]
```

Example 195 (unknown):
```unknown
Epoch 45/500:   9%|▉         | 45/500 [05:18<52:33,  6.93s/it, loss=1.52e+04, v_num=1]
```

Example 196 (unknown):
```unknown
Epoch 45/500:   9%|▉         | 45/500 [05:18<52:33,  6.93s/it, loss=1.54e+04, v_num=1]
```

Example 197 (unknown):
```unknown
Epoch 46/500:   9%|▉         | 45/500 [05:18<52:33,  6.93s/it, loss=1.54e+04, v_num=1]
```

Example 198 (unknown):
```unknown
Epoch 46/500:   9%|▉         | 46/500 [05:25<52:17,  6.91s/it, loss=1.54e+04, v_num=1]
```

Example 199 (unknown):
```unknown
Epoch 46/500:   9%|▉         | 46/500 [05:25<52:17,  6.91s/it, loss=1.55e+04, v_num=1]
```

Example 200 (unknown):
```unknown
Epoch 47/500:   9%|▉         | 46/500 [05:25<52:17,  6.91s/it, loss=1.55e+04, v_num=1]
```

Example 201 (unknown):
```unknown
Epoch 47/500:   9%|▉         | 47/500 [05:32<52:17,  6.93s/it, loss=1.55e+04, v_num=1]
```

Example 202 (unknown):
```unknown
Epoch 47/500:   9%|▉         | 47/500 [05:32<52:17,  6.93s/it, loss=1.54e+04, v_num=1]
```

Example 203 (unknown):
```unknown
Epoch 48/500:   9%|▉         | 47/500 [05:32<52:17,  6.93s/it, loss=1.54e+04, v_num=1]
```

Example 204 (unknown):
```unknown
Epoch 48/500:  10%|▉         | 48/500 [05:39<52:03,  6.91s/it, loss=1.54e+04, v_num=1]
```

Example 205 (unknown):
```unknown
Epoch 48/500:  10%|▉         | 48/500 [05:39<52:03,  6.91s/it, loss=1.54e+04, v_num=1]
```

Example 206 (unknown):
```unknown
Epoch 49/500:  10%|▉         | 48/500 [05:39<52:03,  6.91s/it, loss=1.54e+04, v_num=1]
```

Example 207 (unknown):
```unknown
Epoch 49/500:  10%|▉         | 49/500 [05:46<51:31,  6.85s/it, loss=1.54e+04, v_num=1]
```

Example 208 (unknown):
```unknown
Epoch 49/500:  10%|▉         | 49/500 [05:46<51:31,  6.85s/it, loss=1.53e+04, v_num=1]
```

Example 209 (unknown):
```unknown
Epoch 50/500:  10%|▉         | 49/500 [05:46<51:31,  6.85s/it, loss=1.53e+04, v_num=1]
```

Example 210 (unknown):
```unknown
Epoch 50/500:  10%|█         | 50/500 [05:53<51:32,  6.87s/it, loss=1.53e+04, v_num=1]
```

Example 211 (unknown):
```unknown
Epoch 50/500:  10%|█         | 50/500 [05:53<51:32,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 212 (unknown):
```unknown
Epoch 51/500:  10%|█         | 50/500 [05:53<51:32,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 213 (unknown):
```unknown
Epoch 51/500:  10%|█         | 51/500 [05:59<51:35,  6.89s/it, loss=1.54e+04, v_num=1]
```

Example 214 (unknown):
```unknown
Epoch 51/500:  10%|█         | 51/500 [06:00<51:35,  6.89s/it, loss=1.53e+04, v_num=1]
```

Example 215 (unknown):
```unknown
Epoch 52/500:  10%|█         | 51/500 [06:00<51:35,  6.89s/it, loss=1.53e+04, v_num=1]
```

Example 216 (unknown):
```unknown
Epoch 52/500:  10%|█         | 52/500 [06:06<51:39,  6.92s/it, loss=1.53e+04, v_num=1]
```

Example 217 (unknown):
```unknown
Epoch 52/500:  10%|█         | 52/500 [06:06<51:39,  6.92s/it, loss=1.54e+04, v_num=1]
```

Example 218 (unknown):
```unknown
Epoch 53/500:  10%|█         | 52/500 [06:06<51:39,  6.92s/it, loss=1.54e+04, v_num=1]
```

Example 219 (unknown):
```unknown
Epoch 53/500:  11%|█         | 53/500 [06:13<50:53,  6.83s/it, loss=1.54e+04, v_num=1]
```

Example 220 (unknown):
```unknown
Epoch 53/500:  11%|█         | 53/500 [06:13<50:53,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 221 (unknown):
```unknown
Epoch 54/500:  11%|█         | 53/500 [06:13<50:53,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 222 (unknown):
```unknown
Epoch 54/500:  11%|█         | 54/500 [06:20<50:20,  6.77s/it, loss=1.51e+04, v_num=1]
```

Example 223 (unknown):
```unknown
Epoch 54/500:  11%|█         | 54/500 [06:20<50:20,  6.77s/it, loss=1.54e+04, v_num=1]
```

Example 224 (unknown):
```unknown
Epoch 55/500:  11%|█         | 54/500 [06:20<50:20,  6.77s/it, loss=1.54e+04, v_num=1]
```

Example 225 (unknown):
```unknown
Epoch 55/500:  11%|█         | 55/500 [06:27<50:13,  6.77s/it, loss=1.54e+04, v_num=1]
```

Example 226 (unknown):
```unknown
Epoch 55/500:  11%|█         | 55/500 [06:27<50:13,  6.77s/it, loss=1.55e+04, v_num=1]
```

Example 227 (unknown):
```unknown
Epoch 56/500:  11%|█         | 55/500 [06:27<50:13,  6.77s/it, loss=1.55e+04, v_num=1]
```

Example 228 (unknown):
```unknown
Epoch 56/500:  11%|█         | 56/500 [06:33<50:14,  6.79s/it, loss=1.55e+04, v_num=1]
```

Example 229 (unknown):
```unknown
Epoch 56/500:  11%|█         | 56/500 [06:33<50:14,  6.79s/it, loss=1.52e+04, v_num=1]
```

Example 230 (unknown):
```unknown
Epoch 57/500:  11%|█         | 56/500 [06:33<50:14,  6.79s/it, loss=1.52e+04, v_num=1]
```

Example 231 (unknown):
```unknown
Epoch 57/500:  11%|█▏        | 57/500 [06:40<50:39,  6.86s/it, loss=1.52e+04, v_num=1]
```

Example 232 (unknown):
```unknown
Epoch 57/500:  11%|█▏        | 57/500 [06:40<50:39,  6.86s/it, loss=1.52e+04, v_num=1]
```

Example 233 (unknown):
```unknown
Epoch 58/500:  11%|█▏        | 57/500 [06:40<50:39,  6.86s/it, loss=1.52e+04, v_num=1]
```

Example 234 (unknown):
```unknown
Epoch 58/500:  12%|█▏        | 58/500 [06:47<50:46,  6.89s/it, loss=1.52e+04, v_num=1]
```

Example 235 (unknown):
```unknown
Epoch 58/500:  12%|█▏        | 58/500 [06:47<50:46,  6.89s/it, loss=1.56e+04, v_num=1]
```

Example 236 (unknown):
```unknown
Epoch 59/500:  12%|█▏        | 58/500 [06:47<50:46,  6.89s/it, loss=1.56e+04, v_num=1]
```

Example 237 (unknown):
```unknown
Epoch 59/500:  12%|█▏        | 59/500 [06:54<49:50,  6.78s/it, loss=1.56e+04, v_num=1]
```

Example 238 (unknown):
```unknown
Epoch 59/500:  12%|█▏        | 59/500 [06:54<49:50,  6.78s/it, loss=1.57e+04, v_num=1]
```

Example 239 (unknown):
```unknown
Epoch 60/500:  12%|█▏        | 59/500 [06:54<49:50,  6.78s/it, loss=1.57e+04, v_num=1]
```

Example 240 (unknown):
```unknown
Epoch 60/500:  12%|█▏        | 60/500 [07:01<50:12,  6.85s/it, loss=1.57e+04, v_num=1]
```

Example 241 (unknown):
```unknown
Epoch 60/500:  12%|█▏        | 60/500 [07:01<50:12,  6.85s/it, loss=1.54e+04, v_num=1]
```

Example 242 (unknown):
```unknown
Epoch 61/500:  12%|█▏        | 60/500 [07:01<50:12,  6.85s/it, loss=1.54e+04, v_num=1]
```

Example 243 (unknown):
```unknown
Epoch 61/500:  12%|█▏        | 61/500 [07:08<50:16,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 244 (unknown):
```unknown
Epoch 61/500:  12%|█▏        | 61/500 [07:08<50:16,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 245 (unknown):
```unknown
Epoch 62/500:  12%|█▏        | 61/500 [07:08<50:16,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 246 (unknown):
```unknown
Epoch 62/500:  12%|█▏        | 62/500 [07:15<49:53,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 247 (unknown):
```unknown
Epoch 62/500:  12%|█▏        | 62/500 [07:15<49:53,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 248 (unknown):
```unknown
Epoch 63/500:  12%|█▏        | 62/500 [07:15<49:53,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 249 (unknown):
```unknown
Epoch 63/500:  13%|█▎        | 63/500 [07:22<50:12,  6.89s/it, loss=1.52e+04, v_num=1]
```

Example 250 (unknown):
```unknown
Epoch 63/500:  13%|█▎        | 63/500 [07:22<50:12,  6.89s/it, loss=1.54e+04, v_num=1]
```

Example 251 (unknown):
```unknown
Epoch 64/500:  13%|█▎        | 63/500 [07:22<50:12,  6.89s/it, loss=1.54e+04, v_num=1]
```

Example 252 (unknown):
```unknown
Epoch 64/500:  13%|█▎        | 64/500 [07:28<49:46,  6.85s/it, loss=1.54e+04, v_num=1]
```

Example 253 (unknown):
```unknown
Epoch 64/500:  13%|█▎        | 64/500 [07:28<49:46,  6.85s/it, loss=1.54e+04, v_num=1]
```

Example 254 (unknown):
```unknown
Epoch 65/500:  13%|█▎        | 64/500 [07:28<49:46,  6.85s/it, loss=1.54e+04, v_num=1]
```

Example 255 (unknown):
```unknown
Epoch 65/500:  13%|█▎        | 65/500 [07:35<49:37,  6.84s/it, loss=1.54e+04, v_num=1]
```

Example 256 (unknown):
```unknown
Epoch 65/500:  13%|█▎        | 65/500 [07:35<49:37,  6.84s/it, loss=1.56e+04, v_num=1]
```

Example 257 (unknown):
```unknown
Epoch 66/500:  13%|█▎        | 65/500 [07:35<49:37,  6.84s/it, loss=1.56e+04, v_num=1]
```

Example 258 (unknown):
```unknown
Epoch 66/500:  13%|█▎        | 66/500 [07:42<49:48,  6.88s/it, loss=1.56e+04, v_num=1]
```

Example 259 (unknown):
```unknown
Epoch 66/500:  13%|█▎        | 66/500 [07:42<49:48,  6.88s/it, loss=1.5e+04, v_num=1]
```

Example 260 (unknown):
```unknown
Epoch 67/500:  13%|█▎        | 66/500 [07:42<49:48,  6.88s/it, loss=1.5e+04, v_num=1]
```

Example 261 (unknown):
```unknown
Epoch 67/500:  13%|█▎        | 67/500 [07:49<49:52,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 262 (unknown):
```unknown
Epoch 67/500:  13%|█▎        | 67/500 [07:49<49:52,  6.91s/it, loss=1.53e+04, v_num=1]
```

Example 263 (unknown):
```unknown
Epoch 68/500:  13%|█▎        | 67/500 [07:49<49:52,  6.91s/it, loss=1.53e+04, v_num=1]
```

Example 264 (unknown):
```unknown
Epoch 68/500:  14%|█▎        | 68/500 [07:56<49:22,  6.86s/it, loss=1.53e+04, v_num=1]
```

Example 265 (unknown):
```unknown
Epoch 68/500:  14%|█▎        | 68/500 [07:56<49:22,  6.86s/it, loss=1.55e+04, v_num=1]
```

Example 266 (unknown):
```unknown
Epoch 69/500:  14%|█▎        | 68/500 [07:56<49:22,  6.86s/it, loss=1.55e+04, v_num=1]
```

Example 267 (unknown):
```unknown
Epoch 69/500:  14%|█▍        | 69/500 [08:03<49:19,  6.87s/it, loss=1.55e+04, v_num=1]
```

Example 268 (unknown):
```unknown
Epoch 69/500:  14%|█▍        | 69/500 [08:03<49:19,  6.87s/it, loss=1.5e+04, v_num=1]
```

Example 269 (unknown):
```unknown
Epoch 70/500:  14%|█▍        | 69/500 [08:03<49:19,  6.87s/it, loss=1.5e+04, v_num=1]
```

Example 270 (unknown):
```unknown
Epoch 70/500:  14%|█▍        | 70/500 [08:10<49:21,  6.89s/it, loss=1.5e+04, v_num=1]
```

Example 271 (unknown):
```unknown
Epoch 70/500:  14%|█▍        | 70/500 [08:10<49:21,  6.89s/it, loss=1.56e+04, v_num=1]
```

Example 272 (unknown):
```unknown
Epoch 71/500:  14%|█▍        | 70/500 [08:10<49:21,  6.89s/it, loss=1.56e+04, v_num=1]
```

Example 273 (unknown):
```unknown
Epoch 71/500:  14%|█▍        | 71/500 [08:17<49:25,  6.91s/it, loss=1.56e+04, v_num=1]
```

Example 274 (unknown):
```unknown
Epoch 71/500:  14%|█▍        | 71/500 [08:17<49:25,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 275 (unknown):
```unknown
Epoch 72/500:  14%|█▍        | 71/500 [08:17<49:25,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 276 (unknown):
```unknown
Epoch 72/500:  14%|█▍        | 72/500 [08:23<49:07,  6.89s/it, loss=1.52e+04, v_num=1]
```

Example 277 (unknown):
```unknown
Epoch 72/500:  14%|█▍        | 72/500 [08:23<49:07,  6.89s/it, loss=1.52e+04, v_num=1]
```

Example 278 (unknown):
```unknown
Epoch 73/500:  14%|█▍        | 72/500 [08:23<49:07,  6.89s/it, loss=1.52e+04, v_num=1]
```

Example 279 (unknown):
```unknown
Epoch 73/500:  15%|█▍        | 73/500 [08:30<48:26,  6.81s/it, loss=1.52e+04, v_num=1]
```

Example 280 (unknown):
```unknown
Epoch 73/500:  15%|█▍        | 73/500 [08:30<48:26,  6.81s/it, loss=1.53e+04, v_num=1]
```

Example 281 (unknown):
```unknown
Epoch 74/500:  15%|█▍        | 73/500 [08:30<48:26,  6.81s/it, loss=1.53e+04, v_num=1]
```

Example 282 (unknown):
```unknown
Epoch 74/500:  15%|█▍        | 74/500 [08:37<48:35,  6.84s/it, loss=1.53e+04, v_num=1]
```

Example 283 (unknown):
```unknown
Epoch 74/500:  15%|█▍        | 74/500 [08:37<48:35,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 284 (unknown):
```unknown
Epoch 75/500:  15%|█▍        | 74/500 [08:37<48:35,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 285 (unknown):
```unknown
Epoch 75/500:  15%|█▌        | 75/500 [08:44<48:16,  6.82s/it, loss=1.51e+04, v_num=1]
```

Example 286 (unknown):
```unknown
Epoch 75/500:  15%|█▌        | 75/500 [08:44<48:16,  6.82s/it, loss=1.53e+04, v_num=1]
```

Example 287 (unknown):
```unknown
Epoch 76/500:  15%|█▌        | 75/500 [08:44<48:16,  6.82s/it, loss=1.53e+04, v_num=1]
```

Example 288 (unknown):
```unknown
Epoch 76/500:  15%|█▌        | 76/500 [08:50<47:59,  6.79s/it, loss=1.53e+04, v_num=1]
```

Example 289 (unknown):
```unknown
Epoch 76/500:  15%|█▌        | 76/500 [08:50<47:59,  6.79s/it, loss=1.54e+04, v_num=1]
```

Example 290 (unknown):
```unknown
Epoch 77/500:  15%|█▌        | 76/500 [08:50<47:59,  6.79s/it, loss=1.54e+04, v_num=1]
```

Example 291 (unknown):
```unknown
Epoch 77/500:  15%|█▌        | 77/500 [08:57<47:03,  6.67s/it, loss=1.54e+04, v_num=1]
```

Example 292 (unknown):
```unknown
Epoch 77/500:  15%|█▌        | 77/500 [08:57<47:03,  6.67s/it, loss=1.51e+04, v_num=1]
```

Example 293 (unknown):
```unknown
Epoch 78/500:  15%|█▌        | 77/500 [08:57<47:03,  6.67s/it, loss=1.51e+04, v_num=1]
```

Example 294 (unknown):
```unknown
Epoch 78/500:  16%|█▌        | 78/500 [09:04<47:17,  6.72s/it, loss=1.51e+04, v_num=1]
```

Example 295 (unknown):
```unknown
Epoch 78/500:  16%|█▌        | 78/500 [09:04<47:17,  6.72s/it, loss=1.53e+04, v_num=1]
```

Example 296 (unknown):
```unknown
Epoch 79/500:  16%|█▌        | 78/500 [09:04<47:17,  6.72s/it, loss=1.53e+04, v_num=1]
```

Example 297 (unknown):
```unknown
Epoch 79/500:  16%|█▌        | 79/500 [09:10<47:12,  6.73s/it, loss=1.53e+04, v_num=1]
```

Example 298 (unknown):
```unknown
Epoch 79/500:  16%|█▌        | 79/500 [09:10<47:12,  6.73s/it, loss=1.53e+04, v_num=1]
```

Example 299 (unknown):
```unknown
Epoch 80/500:  16%|█▌        | 79/500 [09:10<47:12,  6.73s/it, loss=1.53e+04, v_num=1]
```

Example 300 (unknown):
```unknown
Epoch 80/500:  16%|█▌        | 80/500 [09:17<46:25,  6.63s/it, loss=1.53e+04, v_num=1]
```

Example 301 (unknown):
```unknown
Epoch 80/500:  16%|█▌        | 80/500 [09:17<46:25,  6.63s/it, loss=1.54e+04, v_num=1]
```

Example 302 (unknown):
```unknown
Epoch 81/500:  16%|█▌        | 80/500 [09:17<46:25,  6.63s/it, loss=1.54e+04, v_num=1]
```

Example 303 (unknown):
```unknown
Epoch 81/500:  16%|█▌        | 81/500 [09:23<46:00,  6.59s/it, loss=1.54e+04, v_num=1]
```

Example 304 (unknown):
```unknown
Epoch 81/500:  16%|█▌        | 81/500 [09:23<46:00,  6.59s/it, loss=1.53e+04, v_num=1]
```

Example 305 (unknown):
```unknown
Epoch 82/500:  16%|█▌        | 81/500 [09:23<46:00,  6.59s/it, loss=1.53e+04, v_num=1]
```

Example 306 (unknown):
```unknown
Epoch 82/500:  16%|█▋        | 82/500 [09:30<45:25,  6.52s/it, loss=1.53e+04, v_num=1]
```

Example 307 (unknown):
```unknown
Epoch 82/500:  16%|█▋        | 82/500 [09:30<45:25,  6.52s/it, loss=1.55e+04, v_num=1]
```

Example 308 (unknown):
```unknown
Epoch 83/500:  16%|█▋        | 82/500 [09:30<45:25,  6.52s/it, loss=1.55e+04, v_num=1]
```

Example 309 (unknown):
```unknown
Epoch 83/500:  17%|█▋        | 83/500 [09:36<45:49,  6.59s/it, loss=1.55e+04, v_num=1]
```

Example 310 (unknown):
```unknown
Epoch 83/500:  17%|█▋        | 83/500 [09:36<45:49,  6.59s/it, loss=1.57e+04, v_num=1]
```

Example 311 (unknown):
```unknown
Epoch 84/500:  17%|█▋        | 83/500 [09:36<45:49,  6.59s/it, loss=1.57e+04, v_num=1]
```

Example 312 (unknown):
```unknown
Epoch 84/500:  17%|█▋        | 84/500 [09:43<46:03,  6.64s/it, loss=1.57e+04, v_num=1]
```

Example 313 (unknown):
```unknown
Epoch 84/500:  17%|█▋        | 84/500 [09:43<46:03,  6.64s/it, loss=1.51e+04, v_num=1]
```

Example 314 (unknown):
```unknown
Epoch 85/500:  17%|█▋        | 84/500 [09:43<46:03,  6.64s/it, loss=1.51e+04, v_num=1]
```

Example 315 (unknown):
```unknown
Epoch 85/500:  17%|█▋        | 85/500 [09:50<46:38,  6.74s/it, loss=1.51e+04, v_num=1]
```

Example 316 (unknown):
```unknown
Epoch 85/500:  17%|█▋        | 85/500 [09:50<46:38,  6.74s/it, loss=1.54e+04, v_num=1]
```

Example 317 (unknown):
```unknown
Epoch 86/500:  17%|█▋        | 85/500 [09:50<46:38,  6.74s/it, loss=1.54e+04, v_num=1]
```

Example 318 (unknown):
```unknown
Epoch 86/500:  17%|█▋        | 86/500 [09:57<47:00,  6.81s/it, loss=1.54e+04, v_num=1]
```

Example 319 (unknown):
```unknown
Epoch 86/500:  17%|█▋        | 86/500 [09:57<47:00,  6.81s/it, loss=1.53e+04, v_num=1]
```

Example 320 (unknown):
```unknown
Epoch 87/500:  17%|█▋        | 86/500 [09:57<47:00,  6.81s/it, loss=1.53e+04, v_num=1]
```

Example 321 (unknown):
```unknown
Epoch 87/500:  17%|█▋        | 87/500 [10:04<46:55,  6.82s/it, loss=1.53e+04, v_num=1]
```

Example 322 (unknown):
```unknown
Epoch 87/500:  17%|█▋        | 87/500 [10:04<46:55,  6.82s/it, loss=1.51e+04, v_num=1]
```

Example 323 (unknown):
```unknown
Epoch 88/500:  17%|█▋        | 87/500 [10:04<46:55,  6.82s/it, loss=1.51e+04, v_num=1]
```

Example 324 (unknown):
```unknown
Epoch 88/500:  18%|█▊        | 88/500 [10:11<46:19,  6.75s/it, loss=1.51e+04, v_num=1]
```

Example 325 (unknown):
```unknown
Epoch 88/500:  18%|█▊        | 88/500 [10:11<46:19,  6.75s/it, loss=1.52e+04, v_num=1]
```

Example 326 (unknown):
```unknown
Epoch 89/500:  18%|█▊        | 88/500 [10:11<46:19,  6.75s/it, loss=1.52e+04, v_num=1]
```

Example 327 (unknown):
```unknown
Epoch 89/500:  18%|█▊        | 89/500 [10:17<45:07,  6.59s/it, loss=1.52e+04, v_num=1]
```

Example 328 (unknown):
```unknown
Epoch 89/500:  18%|█▊        | 89/500 [10:17<45:07,  6.59s/it, loss=1.51e+04, v_num=1]
```

Example 329 (unknown):
```unknown
Epoch 90/500:  18%|█▊        | 89/500 [10:17<45:07,  6.59s/it, loss=1.51e+04, v_num=1]
```

Example 330 (unknown):
```unknown
Epoch 90/500:  18%|█▊        | 90/500 [10:24<45:23,  6.64s/it, loss=1.51e+04, v_num=1]
```

Example 331 (unknown):
```unknown
Epoch 90/500:  18%|█▊        | 90/500 [10:24<45:23,  6.64s/it, loss=1.54e+04, v_num=1]
```

Example 332 (unknown):
```unknown
Epoch 91/500:  18%|█▊        | 90/500 [10:24<45:23,  6.64s/it, loss=1.54e+04, v_num=1]
```

Example 333 (unknown):
```unknown
Epoch 91/500:  18%|█▊        | 91/500 [10:30<45:44,  6.71s/it, loss=1.54e+04, v_num=1]
```

Example 334 (unknown):
```unknown
Epoch 91/500:  18%|█▊        | 91/500 [10:30<45:44,  6.71s/it, loss=1.53e+04, v_num=1]
```

Example 335 (unknown):
```unknown
Epoch 92/500:  18%|█▊        | 91/500 [10:30<45:44,  6.71s/it, loss=1.53e+04, v_num=1]
```

Example 336 (unknown):
```unknown
Epoch 92/500:  18%|█▊        | 92/500 [10:37<45:56,  6.76s/it, loss=1.53e+04, v_num=1]
```

Example 337 (unknown):
```unknown
Epoch 92/500:  18%|█▊        | 92/500 [10:37<45:56,  6.76s/it, loss=1.51e+04, v_num=1]
```

Example 338 (unknown):
```unknown
Epoch 93/500:  18%|█▊        | 92/500 [10:37<45:56,  6.76s/it, loss=1.51e+04, v_num=1]
```

Example 339 (unknown):
```unknown
Epoch 93/500:  19%|█▊        | 93/500 [10:44<46:11,  6.81s/it, loss=1.51e+04, v_num=1]
```

Example 340 (unknown):
```unknown
Epoch 93/500:  19%|█▊        | 93/500 [10:44<46:11,  6.81s/it, loss=1.53e+04, v_num=1]
```

Example 341 (unknown):
```unknown
Epoch 94/500:  19%|█▊        | 93/500 [10:44<46:11,  6.81s/it, loss=1.53e+04, v_num=1]
```

Example 342 (unknown):
```unknown
Epoch 94/500:  19%|█▉        | 94/500 [10:51<45:41,  6.75s/it, loss=1.53e+04, v_num=1]
```

Example 343 (unknown):
```unknown
Epoch 94/500:  19%|█▉        | 94/500 [10:51<45:41,  6.75s/it, loss=1.54e+04, v_num=1]
```

Example 344 (unknown):
```unknown
Epoch 95/500:  19%|█▉        | 94/500 [10:51<45:41,  6.75s/it, loss=1.54e+04, v_num=1]
```

Example 345 (unknown):
```unknown
Epoch 95/500:  19%|█▉        | 95/500 [10:57<45:02,  6.67s/it, loss=1.54e+04, v_num=1]
```

Example 346 (unknown):
```unknown
Epoch 95/500:  19%|█▉        | 95/500 [10:57<45:02,  6.67s/it, loss=1.53e+04, v_num=1]
```

Example 347 (unknown):
```unknown
Epoch 96/500:  19%|█▉        | 95/500 [10:57<45:02,  6.67s/it, loss=1.53e+04, v_num=1]
```

Example 348 (unknown):
```unknown
Epoch 96/500:  19%|█▉        | 96/500 [11:04<45:05,  6.70s/it, loss=1.53e+04, v_num=1]
```

Example 349 (unknown):
```unknown
Epoch 96/500:  19%|█▉        | 96/500 [11:04<45:05,  6.70s/it, loss=1.49e+04, v_num=1]
```

Example 350 (unknown):
```unknown
Epoch 97/500:  19%|█▉        | 96/500 [11:04<45:05,  6.70s/it, loss=1.49e+04, v_num=1]
```

Example 351 (unknown):
```unknown
Epoch 97/500:  19%|█▉        | 97/500 [11:11<45:26,  6.77s/it, loss=1.49e+04, v_num=1]
```

Example 352 (unknown):
```unknown
Epoch 97/500:  19%|█▉        | 97/500 [11:11<45:26,  6.77s/it, loss=1.53e+04, v_num=1]
```

Example 353 (unknown):
```unknown
Epoch 98/500:  19%|█▉        | 97/500 [11:11<45:26,  6.77s/it, loss=1.53e+04, v_num=1]
```

Example 354 (unknown):
```unknown
Epoch 98/500:  20%|█▉        | 98/500 [11:18<45:20,  6.77s/it, loss=1.53e+04, v_num=1]
```

Example 355 (unknown):
```unknown
Epoch 98/500:  20%|█▉        | 98/500 [11:18<45:20,  6.77s/it, loss=1.54e+04, v_num=1]
```

Example 356 (unknown):
```unknown
Epoch 99/500:  20%|█▉        | 98/500 [11:18<45:20,  6.77s/it, loss=1.54e+04, v_num=1]
```

Example 357 (unknown):
```unknown
Epoch 99/500:  20%|█▉        | 99/500 [11:25<45:34,  6.82s/it, loss=1.54e+04, v_num=1]
```

Example 358 (unknown):
```unknown
Epoch 99/500:  20%|█▉        | 99/500 [11:25<45:34,  6.82s/it, loss=1.55e+04, v_num=1]
```

Example 359 (unknown):
```unknown
Epoch 100/500:  20%|█▉        | 99/500 [11:25<45:34,  6.82s/it, loss=1.55e+04, v_num=1]
```

Example 360 (unknown):
```unknown
Epoch 100/500:  20%|██        | 100/500 [11:32<45:28,  6.82s/it, loss=1.55e+04, v_num=1]
```

Example 361 (unknown):
```unknown
Epoch 100/500:  20%|██        | 100/500 [11:32<45:28,  6.82s/it, loss=1.54e+04, v_num=1]
```

Example 362 (unknown):
```unknown
Epoch 101/500:  20%|██        | 100/500 [11:32<45:28,  6.82s/it, loss=1.54e+04, v_num=1]
```

Example 363 (unknown):
```unknown
Epoch 101/500:  20%|██        | 101/500 [11:39<45:39,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 364 (unknown):
```unknown
Epoch 101/500:  20%|██        | 101/500 [11:39<45:39,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 365 (unknown):
```unknown
Epoch 102/500:  20%|██        | 101/500 [11:39<45:39,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 366 (unknown):
```unknown
Epoch 102/500:  20%|██        | 102/500 [11:46<45:44,  6.90s/it, loss=1.54e+04, v_num=1]
```

Example 367 (unknown):
```unknown
Epoch 102/500:  20%|██        | 102/500 [11:46<45:44,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 368 (unknown):
```unknown
Epoch 103/500:  20%|██        | 102/500 [11:46<45:44,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 369 (unknown):
```unknown
Epoch 103/500:  21%|██        | 103/500 [11:52<45:45,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 370 (unknown):
```unknown
Epoch 103/500:  21%|██        | 103/500 [11:52<45:45,  6.91s/it, loss=1.55e+04, v_num=1]
```

Example 371 (unknown):
```unknown
Epoch 104/500:  21%|██        | 103/500 [11:52<45:45,  6.91s/it, loss=1.55e+04, v_num=1]
```

Example 372 (unknown):
```unknown
Epoch 104/500:  21%|██        | 104/500 [11:59<45:28,  6.89s/it, loss=1.55e+04, v_num=1]
```

Example 373 (unknown):
```unknown
Epoch 104/500:  21%|██        | 104/500 [11:59<45:28,  6.89s/it, loss=1.52e+04, v_num=1]
```

Example 374 (unknown):
```unknown
Epoch 105/500:  21%|██        | 104/500 [11:59<45:28,  6.89s/it, loss=1.52e+04, v_num=1]
```

Example 375 (unknown):
```unknown
Epoch 105/500:  21%|██        | 105/500 [12:06<45:30,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 376 (unknown):
```unknown
Epoch 105/500:  21%|██        | 105/500 [12:06<45:30,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 377 (unknown):
```unknown
Epoch 106/500:  21%|██        | 105/500 [12:06<45:30,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 378 (unknown):
```unknown
Epoch 106/500:  21%|██        | 106/500 [12:13<45:05,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 379 (unknown):
```unknown
Epoch 106/500:  21%|██        | 106/500 [12:13<45:05,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 380 (unknown):
```unknown
Epoch 107/500:  21%|██        | 106/500 [12:13<45:05,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 381 (unknown):
```unknown
Epoch 107/500:  21%|██▏       | 107/500 [12:20<45:03,  6.88s/it, loss=1.54e+04, v_num=1]
```

Example 382 (unknown):
```unknown
Epoch 107/500:  21%|██▏       | 107/500 [12:20<45:03,  6.88s/it, loss=1.54e+04, v_num=1]
```

Example 383 (unknown):
```unknown
Epoch 108/500:  21%|██▏       | 107/500 [12:20<45:03,  6.88s/it, loss=1.54e+04, v_num=1]
```

Example 384 (unknown):
```unknown
Epoch 108/500:  22%|██▏       | 108/500 [12:27<44:56,  6.88s/it, loss=1.54e+04, v_num=1]
```

Example 385 (unknown):
```unknown
Epoch 108/500:  22%|██▏       | 108/500 [12:27<44:56,  6.88s/it, loss=1.53e+04, v_num=1]
```

Example 386 (unknown):
```unknown
Epoch 109/500:  22%|██▏       | 108/500 [12:27<44:56,  6.88s/it, loss=1.53e+04, v_num=1]
```

Example 387 (unknown):
```unknown
Epoch 109/500:  22%|██▏       | 109/500 [12:34<45:03,  6.91s/it, loss=1.53e+04, v_num=1]
```

Example 388 (unknown):
```unknown
Epoch 109/500:  22%|██▏       | 109/500 [12:34<45:03,  6.91s/it, loss=1.53e+04, v_num=1]
```

Example 389 (unknown):
```unknown
Epoch 110/500:  22%|██▏       | 109/500 [12:34<45:03,  6.91s/it, loss=1.53e+04, v_num=1]
```

Example 390 (unknown):
```unknown
Epoch 110/500:  22%|██▏       | 110/500 [12:41<44:58,  6.92s/it, loss=1.53e+04, v_num=1]
```

Example 391 (unknown):
```unknown
Epoch 110/500:  22%|██▏       | 110/500 [12:41<44:58,  6.92s/it, loss=1.53e+04, v_num=1]
```

Example 392 (unknown):
```unknown
Epoch 111/500:  22%|██▏       | 110/500 [12:41<44:58,  6.92s/it, loss=1.53e+04, v_num=1]
```

Example 393 (unknown):
```unknown
Epoch 111/500:  22%|██▏       | 111/500 [12:48<44:43,  6.90s/it, loss=1.53e+04, v_num=1]
```

Example 394 (unknown):
```unknown
Epoch 111/500:  22%|██▏       | 111/500 [12:48<44:43,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 395 (unknown):
```unknown
Epoch 112/500:  22%|██▏       | 111/500 [12:48<44:43,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 396 (unknown):
```unknown
Epoch 112/500:  22%|██▏       | 112/500 [12:55<44:37,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 397 (unknown):
```unknown
Epoch 112/500:  22%|██▏       | 112/500 [12:55<44:37,  6.90s/it, loss=1.54e+04, v_num=1]
```

Example 398 (unknown):
```unknown
Epoch 113/500:  22%|██▏       | 112/500 [12:55<44:37,  6.90s/it, loss=1.54e+04, v_num=1]
```

Example 399 (unknown):
```unknown
Epoch 113/500:  23%|██▎       | 113/500 [13:01<44:35,  6.91s/it, loss=1.54e+04, v_num=1]
```

Example 400 (unknown):
```unknown
Epoch 113/500:  23%|██▎       | 113/500 [13:01<44:35,  6.91s/it, loss=1.55e+04, v_num=1]
```

Example 401 (unknown):
```unknown
Epoch 114/500:  23%|██▎       | 113/500 [13:01<44:35,  6.91s/it, loss=1.55e+04, v_num=1]
```

Example 402 (unknown):
```unknown
Epoch 114/500:  23%|██▎       | 114/500 [13:08<44:35,  6.93s/it, loss=1.55e+04, v_num=1]
```

Example 403 (unknown):
```unknown
Epoch 114/500:  23%|██▎       | 114/500 [13:08<44:35,  6.93s/it, loss=1.5e+04, v_num=1]
```

Example 404 (unknown):
```unknown
Epoch 115/500:  23%|██▎       | 114/500 [13:08<44:35,  6.93s/it, loss=1.5e+04, v_num=1]
```

Example 405 (unknown):
```unknown
Epoch 115/500:  23%|██▎       | 115/500 [13:15<43:32,  6.79s/it, loss=1.5e+04, v_num=1]
```

Example 406 (unknown):
```unknown
Epoch 115/500:  23%|██▎       | 115/500 [13:15<43:32,  6.79s/it, loss=1.52e+04, v_num=1]
```

Example 407 (unknown):
```unknown
Epoch 116/500:  23%|██▎       | 115/500 [13:15<43:32,  6.79s/it, loss=1.52e+04, v_num=1]
```

Example 408 (unknown):
```unknown
Epoch 116/500:  23%|██▎       | 116/500 [13:22<43:51,  6.85s/it, loss=1.52e+04, v_num=1]
```

Example 409 (unknown):
```unknown
Epoch 116/500:  23%|██▎       | 116/500 [13:22<43:51,  6.85s/it, loss=1.54e+04, v_num=1]
```

Example 410 (unknown):
```unknown
Epoch 117/500:  23%|██▎       | 116/500 [13:22<43:51,  6.85s/it, loss=1.54e+04, v_num=1]
```

Example 411 (unknown):
```unknown
Epoch 117/500:  23%|██▎       | 117/500 [13:29<44:14,  6.93s/it, loss=1.54e+04, v_num=1]
```

Example 412 (unknown):
```unknown
Epoch 117/500:  23%|██▎       | 117/500 [13:29<44:14,  6.93s/it, loss=1.51e+04, v_num=1]
```

Example 413 (unknown):
```unknown
Epoch 118/500:  23%|██▎       | 117/500 [13:29<44:14,  6.93s/it, loss=1.51e+04, v_num=1]
```

Example 414 (unknown):
```unknown
Epoch 118/500:  24%|██▎       | 118/500 [13:36<43:55,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 415 (unknown):
```unknown
Epoch 118/500:  24%|██▎       | 118/500 [13:36<43:55,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 416 (unknown):
```unknown
Epoch 119/500:  24%|██▎       | 118/500 [13:36<43:55,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 417 (unknown):
```unknown
Epoch 119/500:  24%|██▍       | 119/500 [13:43<43:48,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 418 (unknown):
```unknown
Epoch 119/500:  24%|██▍       | 119/500 [13:43<43:48,  6.90s/it, loss=1.54e+04, v_num=1]
```

Example 419 (unknown):
```unknown
Epoch 120/500:  24%|██▍       | 119/500 [13:43<43:48,  6.90s/it, loss=1.54e+04, v_num=1]
```

Example 420 (unknown):
```unknown
Epoch 120/500:  24%|██▍       | 120/500 [13:50<43:57,  6.94s/it, loss=1.54e+04, v_num=1]
```

Example 421 (unknown):
```unknown
Epoch 120/500:  24%|██▍       | 120/500 [13:50<43:57,  6.94s/it, loss=1.55e+04, v_num=1]
```

Example 422 (unknown):
```unknown
Epoch 121/500:  24%|██▍       | 120/500 [13:50<43:57,  6.94s/it, loss=1.55e+04, v_num=1]
```

Example 423 (unknown):
```unknown
Epoch 121/500:  24%|██▍       | 121/500 [13:56<43:06,  6.82s/it, loss=1.55e+04, v_num=1]
```

Example 424 (unknown):
```unknown
Epoch 121/500:  24%|██▍       | 121/500 [13:56<43:06,  6.82s/it, loss=1.52e+04, v_num=1]
```

Example 425 (unknown):
```unknown
Epoch 122/500:  24%|██▍       | 121/500 [13:56<43:06,  6.82s/it, loss=1.52e+04, v_num=1]
```

Example 426 (unknown):
```unknown
Epoch 122/500:  24%|██▍       | 122/500 [14:03<43:16,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 427 (unknown):
```unknown
Epoch 122/500:  24%|██▍       | 122/500 [14:03<43:16,  6.87s/it, loss=1.55e+04, v_num=1]
```

Example 428 (unknown):
```unknown
Epoch 123/500:  24%|██▍       | 122/500 [14:03<43:16,  6.87s/it, loss=1.55e+04, v_num=1]
```

Example 429 (unknown):
```unknown
Epoch 123/500:  25%|██▍       | 123/500 [14:10<43:16,  6.89s/it, loss=1.55e+04, v_num=1]
```

Example 430 (unknown):
```unknown
Epoch 123/500:  25%|██▍       | 123/500 [14:10<43:16,  6.89s/it, loss=1.51e+04, v_num=1]
```

Example 431 (unknown):
```unknown
Epoch 124/500:  25%|██▍       | 123/500 [14:10<43:16,  6.89s/it, loss=1.51e+04, v_num=1]
```

Example 432 (unknown):
```unknown
Epoch 124/500:  25%|██▍       | 124/500 [14:17<43:11,  6.89s/it, loss=1.51e+04, v_num=1]
```

Example 433 (unknown):
```unknown
Epoch 124/500:  25%|██▍       | 124/500 [14:17<43:11,  6.89s/it, loss=1.54e+04, v_num=1]
```

Example 434 (unknown):
```unknown
Epoch 125/500:  25%|██▍       | 124/500 [14:17<43:11,  6.89s/it, loss=1.54e+04, v_num=1]
```

Example 435 (unknown):
```unknown
Epoch 125/500:  25%|██▌       | 125/500 [14:24<43:12,  6.91s/it, loss=1.54e+04, v_num=1]
```

Example 436 (unknown):
```unknown
Epoch 125/500:  25%|██▌       | 125/500 [14:24<43:12,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 437 (unknown):
```unknown
Epoch 126/500:  25%|██▌       | 125/500 [14:24<43:12,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 438 (unknown):
```unknown
Epoch 126/500:  25%|██▌       | 126/500 [14:31<42:49,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 439 (unknown):
```unknown
Epoch 126/500:  25%|██▌       | 126/500 [14:31<42:49,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 440 (unknown):
```unknown
Epoch 127/500:  25%|██▌       | 126/500 [14:31<42:49,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 441 (unknown):
```unknown
Epoch 127/500:  25%|██▌       | 127/500 [14:38<42:32,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 442 (unknown):
```unknown
Epoch 127/500:  25%|██▌       | 127/500 [14:38<42:32,  6.84s/it, loss=1.54e+04, v_num=1]
```

Example 443 (unknown):
```unknown
Epoch 128/500:  25%|██▌       | 127/500 [14:38<42:32,  6.84s/it, loss=1.54e+04, v_num=1]
```

Example 444 (unknown):
```unknown
Epoch 128/500:  26%|██▌       | 128/500 [14:45<42:39,  6.88s/it, loss=1.54e+04, v_num=1]
```

Example 445 (unknown):
```unknown
Epoch 128/500:  26%|██▌       | 128/500 [14:45<42:39,  6.88s/it, loss=1.52e+04, v_num=1]
```

Example 446 (unknown):
```unknown
Epoch 129/500:  26%|██▌       | 128/500 [14:45<42:39,  6.88s/it, loss=1.52e+04, v_num=1]
```

Example 447 (unknown):
```unknown
Epoch 129/500:  26%|██▌       | 129/500 [14:52<42:36,  6.89s/it, loss=1.52e+04, v_num=1]
```

Example 448 (unknown):
```unknown
Epoch 129/500:  26%|██▌       | 129/500 [14:52<42:36,  6.89s/it, loss=1.54e+04, v_num=1]
```

Example 449 (unknown):
```unknown
Epoch 130/500:  26%|██▌       | 129/500 [14:52<42:36,  6.89s/it, loss=1.54e+04, v_num=1]
```

Example 450 (unknown):
```unknown
Epoch 130/500:  26%|██▌       | 130/500 [14:58<42:20,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 451 (unknown):
```unknown
Epoch 130/500:  26%|██▌       | 130/500 [14:58<42:20,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 452 (unknown):
```unknown
Epoch 131/500:  26%|██▌       | 130/500 [14:58<42:20,  6.87s/it, loss=1.54e+04, v_num=1]
```

Example 453 (unknown):
```unknown
Epoch 131/500:  26%|██▌       | 131/500 [15:05<41:29,  6.75s/it, loss=1.54e+04, v_num=1]
```

Example 454 (unknown):
```unknown
Epoch 131/500:  26%|██▌       | 131/500 [15:05<41:29,  6.75s/it, loss=1.5e+04, v_num=1]
```

Example 455 (unknown):
```unknown
Epoch 132/500:  26%|██▌       | 131/500 [15:05<41:29,  6.75s/it, loss=1.5e+04, v_num=1]
```

Example 456 (unknown):
```unknown
Epoch 132/500:  26%|██▋       | 132/500 [15:11<41:06,  6.70s/it, loss=1.5e+04, v_num=1]
```

Example 457 (unknown):
```unknown
Epoch 132/500:  26%|██▋       | 132/500 [15:11<41:06,  6.70s/it, loss=1.54e+04, v_num=1]
```

Example 458 (unknown):
```unknown
Epoch 133/500:  26%|██▋       | 132/500 [15:11<41:06,  6.70s/it, loss=1.54e+04, v_num=1]
```

Example 459 (unknown):
```unknown
Epoch 133/500:  27%|██▋       | 133/500 [15:18<40:19,  6.59s/it, loss=1.54e+04, v_num=1]
```

Example 460 (unknown):
```unknown
Epoch 133/500:  27%|██▋       | 133/500 [15:18<40:19,  6.59s/it, loss=1.53e+04, v_num=1]
```

Example 461 (unknown):
```unknown
Epoch 134/500:  27%|██▋       | 133/500 [15:18<40:19,  6.59s/it, loss=1.53e+04, v_num=1]
```

Example 462 (unknown):
```unknown
Epoch 134/500:  27%|██▋       | 134/500 [15:24<40:02,  6.57s/it, loss=1.53e+04, v_num=1]
```

Example 463 (unknown):
```unknown
Epoch 134/500:  27%|██▋       | 134/500 [15:24<40:02,  6.57s/it, loss=1.51e+04, v_num=1]
```

Example 464 (unknown):
```unknown
Epoch 135/500:  27%|██▋       | 134/500 [15:24<40:02,  6.57s/it, loss=1.51e+04, v_num=1]
```

Example 465 (unknown):
```unknown
Epoch 135/500:  27%|██▋       | 135/500 [15:31<39:37,  6.51s/it, loss=1.51e+04, v_num=1]
```

Example 466 (unknown):
```unknown
Epoch 135/500:  27%|██▋       | 135/500 [15:31<39:37,  6.51s/it, loss=1.52e+04, v_num=1]
```

Example 467 (unknown):
```unknown
Epoch 136/500:  27%|██▋       | 135/500 [15:31<39:37,  6.51s/it, loss=1.52e+04, v_num=1]
```

Example 468 (unknown):
```unknown
Epoch 136/500:  27%|██▋       | 136/500 [15:37<40:04,  6.61s/it, loss=1.52e+04, v_num=1]
```

Example 469 (unknown):
```unknown
Epoch 136/500:  27%|██▋       | 136/500 [15:37<40:04,  6.61s/it, loss=1.53e+04, v_num=1]
```

Example 470 (unknown):
```unknown
Epoch 137/500:  27%|██▋       | 136/500 [15:37<40:04,  6.61s/it, loss=1.53e+04, v_num=1]
```

Example 471 (unknown):
```unknown
Epoch 137/500:  27%|██▋       | 137/500 [15:44<40:40,  6.72s/it, loss=1.53e+04, v_num=1]
```

Example 472 (unknown):
```unknown
Epoch 137/500:  27%|██▋       | 137/500 [15:44<40:40,  6.72s/it, loss=1.51e+04, v_num=1]
```

Example 473 (unknown):
```unknown
Epoch 138/500:  27%|██▋       | 137/500 [15:44<40:40,  6.72s/it, loss=1.51e+04, v_num=1]
```

Example 474 (unknown):
```unknown
Epoch 138/500:  28%|██▊       | 138/500 [15:51<40:57,  6.79s/it, loss=1.51e+04, v_num=1]
```

Example 475 (unknown):
```unknown
Epoch 138/500:  28%|██▊       | 138/500 [15:51<40:57,  6.79s/it, loss=1.52e+04, v_num=1]
```

Example 476 (unknown):
```unknown
Epoch 139/500:  28%|██▊       | 138/500 [15:51<40:57,  6.79s/it, loss=1.52e+04, v_num=1]
```

Example 477 (unknown):
```unknown
Epoch 139/500:  28%|██▊       | 139/500 [15:58<40:45,  6.78s/it, loss=1.52e+04, v_num=1]
```

Example 478 (unknown):
```unknown
Epoch 139/500:  28%|██▊       | 139/500 [15:58<40:45,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 479 (unknown):
```unknown
Epoch 140/500:  28%|██▊       | 139/500 [15:58<40:45,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 480 (unknown):
```unknown
Epoch 140/500:  28%|██▊       | 140/500 [16:05<41:00,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 481 (unknown):
```unknown
Epoch 140/500:  28%|██▊       | 140/500 [16:05<41:00,  6.83s/it, loss=1.54e+04, v_num=1]
```

Example 482 (unknown):
```unknown
Epoch 141/500:  28%|██▊       | 140/500 [16:05<41:00,  6.83s/it, loss=1.54e+04, v_num=1]
```

Example 483 (unknown):
```unknown
Epoch 141/500:  28%|██▊       | 141/500 [16:12<40:36,  6.79s/it, loss=1.54e+04, v_num=1]
```

Example 484 (unknown):
```unknown
Epoch 141/500:  28%|██▊       | 141/500 [16:12<40:36,  6.79s/it, loss=1.52e+04, v_num=1]
```

Example 485 (unknown):
```unknown
Epoch 142/500:  28%|██▊       | 141/500 [16:12<40:36,  6.79s/it, loss=1.52e+04, v_num=1]
```

Example 486 (unknown):
```unknown
Epoch 142/500:  28%|██▊       | 142/500 [16:18<39:48,  6.67s/it, loss=1.52e+04, v_num=1]
```

Example 487 (unknown):
```unknown
Epoch 142/500:  28%|██▊       | 142/500 [16:18<39:48,  6.67s/it, loss=1.51e+04, v_num=1]
```

Example 488 (unknown):
```unknown
Epoch 143/500:  28%|██▊       | 142/500 [16:18<39:48,  6.67s/it, loss=1.51e+04, v_num=1]
```

Example 489 (unknown):
```unknown
Epoch 143/500:  29%|██▊       | 143/500 [16:25<39:48,  6.69s/it, loss=1.51e+04, v_num=1]
```

Example 490 (unknown):
```unknown
Epoch 143/500:  29%|██▊       | 143/500 [16:25<39:48,  6.69s/it, loss=1.52e+04, v_num=1]
```

Example 491 (unknown):
```unknown
Epoch 144/500:  29%|██▊       | 143/500 [16:25<39:48,  6.69s/it, loss=1.52e+04, v_num=1]
```

Example 492 (unknown):
```unknown
Epoch 144/500:  29%|██▉       | 144/500 [16:32<39:42,  6.69s/it, loss=1.52e+04, v_num=1]
```

Example 493 (unknown):
```unknown
Epoch 144/500:  29%|██▉       | 144/500 [16:32<39:42,  6.69s/it, loss=1.51e+04, v_num=1]
```

Example 494 (unknown):
```unknown
Epoch 145/500:  29%|██▉       | 144/500 [16:32<39:42,  6.69s/it, loss=1.51e+04, v_num=1]
```

Example 495 (unknown):
```unknown
Epoch 145/500:  29%|██▉       | 145/500 [16:38<39:47,  6.73s/it, loss=1.51e+04, v_num=1]
```

Example 496 (unknown):
```unknown
Epoch 145/500:  29%|██▉       | 145/500 [16:38<39:47,  6.73s/it, loss=1.52e+04, v_num=1]
```

Example 497 (unknown):
```unknown
Epoch 146/500:  29%|██▉       | 145/500 [16:38<39:47,  6.73s/it, loss=1.52e+04, v_num=1]
```

Example 498 (unknown):
```unknown
Epoch 146/500:  29%|██▉       | 146/500 [16:45<40:06,  6.80s/it, loss=1.52e+04, v_num=1]
```

Example 499 (unknown):
```unknown
Epoch 146/500:  29%|██▉       | 146/500 [16:45<40:06,  6.80s/it, loss=1.52e+04, v_num=1]
```

Example 500 (unknown):
```unknown
Epoch 147/500:  29%|██▉       | 146/500 [16:45<40:06,  6.80s/it, loss=1.52e+04, v_num=1]
```

Example 501 (unknown):
```unknown
Epoch 147/500:  29%|██▉       | 147/500 [16:52<39:38,  6.74s/it, loss=1.52e+04, v_num=1]
```

Example 502 (unknown):
```unknown
Epoch 147/500:  29%|██▉       | 147/500 [16:52<39:38,  6.74s/it, loss=1.52e+04, v_num=1]
```

Example 503 (unknown):
```unknown
Epoch 148/500:  29%|██▉       | 147/500 [16:52<39:38,  6.74s/it, loss=1.52e+04, v_num=1]
```

Example 504 (unknown):
```unknown
Epoch 148/500:  30%|██▉       | 148/500 [16:59<39:58,  6.81s/it, loss=1.52e+04, v_num=1]
```

Example 505 (unknown):
```unknown
Epoch 148/500:  30%|██▉       | 148/500 [16:59<39:58,  6.81s/it, loss=1.5e+04, v_num=1]
```

Example 506 (unknown):
```unknown
Epoch 149/500:  30%|██▉       | 148/500 [16:59<39:58,  6.81s/it, loss=1.5e+04, v_num=1]
```

Example 507 (unknown):
```unknown
Epoch 149/500:  30%|██▉       | 149/500 [17:06<40:06,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 508 (unknown):
```unknown
Epoch 149/500:  30%|██▉       | 149/500 [17:06<40:06,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 509 (unknown):
```unknown
Epoch 150/500:  30%|██▉       | 149/500 [17:06<40:06,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 510 (unknown):
```unknown
Epoch 150/500:  30%|███       | 150/500 [17:13<40:17,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 511 (unknown):
```unknown
Epoch 150/500:  30%|███       | 150/500 [17:13<40:17,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 512 (unknown):
```unknown
Epoch 151/500:  30%|███       | 150/500 [17:13<40:17,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 513 (unknown):
```unknown
Epoch 151/500:  30%|███       | 151/500 [17:20<40:17,  6.93s/it, loss=1.5e+04, v_num=1]
```

Example 514 (unknown):
```unknown
Epoch 151/500:  30%|███       | 151/500 [17:20<40:17,  6.93s/it, loss=1.54e+04, v_num=1]
```

Example 515 (unknown):
```unknown
Epoch 152/500:  30%|███       | 151/500 [17:20<40:17,  6.93s/it, loss=1.54e+04, v_num=1]
```

Example 516 (unknown):
```unknown
Epoch 152/500:  30%|███       | 152/500 [17:27<39:53,  6.88s/it, loss=1.54e+04, v_num=1]
```

Example 517 (unknown):
```unknown
Epoch 152/500:  30%|███       | 152/500 [17:27<39:53,  6.88s/it, loss=1.52e+04, v_num=1]
```

Example 518 (unknown):
```unknown
Epoch 153/500:  30%|███       | 152/500 [17:27<39:53,  6.88s/it, loss=1.52e+04, v_num=1]
```

Example 519 (unknown):
```unknown
Epoch 153/500:  31%|███       | 153/500 [17:33<39:32,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 520 (unknown):
```unknown
Epoch 153/500:  31%|███       | 153/500 [17:33<39:32,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 521 (unknown):
```unknown
Epoch 154/500:  31%|███       | 153/500 [17:33<39:32,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 522 (unknown):
```unknown
Epoch 154/500:  31%|███       | 154/500 [17:40<39:37,  6.87s/it, loss=1.49e+04, v_num=1]
```

Example 523 (unknown):
```unknown
Epoch 154/500:  31%|███       | 154/500 [17:40<39:37,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 524 (unknown):
```unknown
Epoch 155/500:  31%|███       | 154/500 [17:40<39:37,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 525 (unknown):
```unknown
Epoch 155/500:  31%|███       | 155/500 [17:47<39:27,  6.86s/it, loss=1.52e+04, v_num=1]
```

Example 526 (unknown):
```unknown
Epoch 155/500:  31%|███       | 155/500 [17:47<39:27,  6.86s/it, loss=1.53e+04, v_num=1]
```

Example 527 (unknown):
```unknown
Epoch 156/500:  31%|███       | 155/500 [17:47<39:27,  6.86s/it, loss=1.53e+04, v_num=1]
```

Example 528 (unknown):
```unknown
Epoch 156/500:  31%|███       | 156/500 [17:54<39:22,  6.87s/it, loss=1.53e+04, v_num=1]
```

Example 529 (unknown):
```unknown
Epoch 156/500:  31%|███       | 156/500 [17:54<39:22,  6.87s/it, loss=1.56e+04, v_num=1]
```

Example 530 (unknown):
```unknown
Epoch 157/500:  31%|███       | 156/500 [17:54<39:22,  6.87s/it, loss=1.56e+04, v_num=1]
```

Example 531 (unknown):
```unknown
Epoch 157/500:  31%|███▏      | 157/500 [18:01<39:28,  6.90s/it, loss=1.56e+04, v_num=1]
```

Example 532 (unknown):
```unknown
Epoch 157/500:  31%|███▏      | 157/500 [18:01<39:28,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 533 (unknown):
```unknown
Epoch 158/500:  31%|███▏      | 157/500 [18:01<39:28,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 534 (unknown):
```unknown
Epoch 158/500:  32%|███▏      | 158/500 [18:08<39:08,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 535 (unknown):
```unknown
Epoch 158/500:  32%|███▏      | 158/500 [18:08<39:08,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 536 (unknown):
```unknown
Epoch 159/500:  32%|███▏      | 158/500 [18:08<39:08,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 537 (unknown):
```unknown
Epoch 159/500:  32%|███▏      | 159/500 [18:15<38:45,  6.82s/it, loss=1.51e+04, v_num=1]
```

Example 538 (unknown):
```unknown
Epoch 159/500:  32%|███▏      | 159/500 [18:15<38:45,  6.82s/it, loss=1.5e+04, v_num=1]
```

Example 539 (unknown):
```unknown
Epoch 160/500:  32%|███▏      | 159/500 [18:15<38:45,  6.82s/it, loss=1.5e+04, v_num=1]
```

Example 540 (unknown):
```unknown
Epoch 160/500:  32%|███▏      | 160/500 [18:21<38:30,  6.80s/it, loss=1.5e+04, v_num=1]
```

Example 541 (unknown):
```unknown
Epoch 160/500:  32%|███▏      | 160/500 [18:21<38:30,  6.80s/it, loss=1.51e+04, v_num=1]
```

Example 542 (unknown):
```unknown
Epoch 161/500:  32%|███▏      | 160/500 [18:21<38:30,  6.80s/it, loss=1.51e+04, v_num=1]
```

Example 543 (unknown):
```unknown
Epoch 161/500:  32%|███▏      | 161/500 [18:28<38:36,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 544 (unknown):
```unknown
Epoch 161/500:  32%|███▏      | 161/500 [18:28<38:36,  6.83s/it, loss=1.52e+04, v_num=1]
```

Example 545 (unknown):
```unknown
Epoch 162/500:  32%|███▏      | 161/500 [18:28<38:36,  6.83s/it, loss=1.52e+04, v_num=1]
```

Example 546 (unknown):
```unknown
Epoch 162/500:  32%|███▏      | 162/500 [18:35<38:45,  6.88s/it, loss=1.52e+04, v_num=1]
```

Example 547 (unknown):
```unknown
Epoch 162/500:  32%|███▏      | 162/500 [18:35<38:45,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 548 (unknown):
```unknown
Epoch 163/500:  32%|███▏      | 162/500 [18:35<38:45,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 549 (unknown):
```unknown
Epoch 163/500:  33%|███▎      | 163/500 [18:42<38:46,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 550 (unknown):
```unknown
Epoch 163/500:  33%|███▎      | 163/500 [18:42<38:46,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 551 (unknown):
```unknown
Epoch 164/500:  33%|███▎      | 163/500 [18:42<38:46,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 552 (unknown):
```unknown
Epoch 164/500:  33%|███▎      | 164/500 [18:49<38:43,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 553 (unknown):
```unknown
Epoch 164/500:  33%|███▎      | 164/500 [18:49<38:43,  6.92s/it, loss=1.54e+04, v_num=1]
```

Example 554 (unknown):
```unknown
Epoch 165/500:  33%|███▎      | 164/500 [18:49<38:43,  6.92s/it, loss=1.54e+04, v_num=1]
```

Example 555 (unknown):
```unknown
Epoch 165/500:  33%|███▎      | 165/500 [18:56<38:18,  6.86s/it, loss=1.54e+04, v_num=1]
```

Example 556 (unknown):
```unknown
Epoch 165/500:  33%|███▎      | 165/500 [18:56<38:18,  6.86s/it, loss=1.55e+04, v_num=1]
```

Example 557 (unknown):
```unknown
Epoch 166/500:  33%|███▎      | 165/500 [18:56<38:18,  6.86s/it, loss=1.55e+04, v_num=1]
```

Example 558 (unknown):
```unknown
Epoch 166/500:  33%|███▎      | 166/500 [19:03<38:19,  6.88s/it, loss=1.55e+04, v_num=1]
```

Example 559 (unknown):
```unknown
Epoch 166/500:  33%|███▎      | 166/500 [19:03<38:19,  6.88s/it, loss=1.56e+04, v_num=1]
```

Example 560 (unknown):
```unknown
Epoch 167/500:  33%|███▎      | 166/500 [19:03<38:19,  6.88s/it, loss=1.56e+04, v_num=1]
```

Example 561 (unknown):
```unknown
Epoch 167/500:  33%|███▎      | 167/500 [19:10<37:57,  6.84s/it, loss=1.56e+04, v_num=1]
```

Example 562 (unknown):
```unknown
Epoch 167/500:  33%|███▎      | 167/500 [19:10<37:57,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 563 (unknown):
```unknown
Epoch 168/500:  33%|███▎      | 167/500 [19:10<37:57,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 564 (unknown):
```unknown
Epoch 168/500:  34%|███▎      | 168/500 [19:17<38:11,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 565 (unknown):
```unknown
Epoch 168/500:  34%|███▎      | 168/500 [19:17<38:11,  6.90s/it, loss=1.54e+04, v_num=1]
```

Example 566 (unknown):
```unknown
Epoch 169/500:  34%|███▎      | 168/500 [19:17<38:11,  6.90s/it, loss=1.54e+04, v_num=1]
```

Example 567 (unknown):
```unknown
Epoch 169/500:  34%|███▍      | 169/500 [19:23<38:06,  6.91s/it, loss=1.54e+04, v_num=1]
```

Example 568 (unknown):
```unknown
Epoch 169/500:  34%|███▍      | 169/500 [19:23<38:06,  6.91s/it, loss=1.51e+04, v_num=1]
```

Example 569 (unknown):
```unknown
Epoch 170/500:  34%|███▍      | 169/500 [19:23<38:06,  6.91s/it, loss=1.51e+04, v_num=1]
```

Example 570 (unknown):
```unknown
Epoch 170/500:  34%|███▍      | 170/500 [19:30<38:05,  6.93s/it, loss=1.51e+04, v_num=1]
```

Example 571 (unknown):
```unknown
Epoch 170/500:  34%|███▍      | 170/500 [19:30<38:05,  6.93s/it, loss=1.53e+04, v_num=1]
```

Example 572 (unknown):
```unknown
Epoch 171/500:  34%|███▍      | 170/500 [19:30<38:05,  6.93s/it, loss=1.53e+04, v_num=1]
```

Example 573 (unknown):
```unknown
Epoch 171/500:  34%|███▍      | 171/500 [19:37<38:01,  6.94s/it, loss=1.53e+04, v_num=1]
```

Example 574 (unknown):
```unknown
Epoch 171/500:  34%|███▍      | 171/500 [19:37<38:01,  6.94s/it, loss=1.52e+04, v_num=1]
```

Example 575 (unknown):
```unknown
Epoch 172/500:  34%|███▍      | 171/500 [19:37<38:01,  6.94s/it, loss=1.52e+04, v_num=1]
```

Example 576 (unknown):
```unknown
Epoch 172/500:  34%|███▍      | 172/500 [19:44<37:43,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 577 (unknown):
```unknown
Epoch 172/500:  34%|███▍      | 172/500 [19:44<37:43,  6.90s/it, loss=1.49e+04, v_num=1]
```

Example 578 (unknown):
```unknown
Epoch 173/500:  34%|███▍      | 172/500 [19:44<37:43,  6.90s/it, loss=1.49e+04, v_num=1]
```

Example 579 (unknown):
```unknown
Epoch 173/500:  35%|███▍      | 173/500 [19:51<37:44,  6.93s/it, loss=1.49e+04, v_num=1]
```

Example 580 (unknown):
```unknown
Epoch 173/500:  35%|███▍      | 173/500 [19:51<37:44,  6.93s/it, loss=1.48e+04, v_num=1]
```

Example 581 (unknown):
```unknown
Epoch 174/500:  35%|███▍      | 173/500 [19:51<37:44,  6.93s/it, loss=1.48e+04, v_num=1]
```

Example 582 (unknown):
```unknown
Epoch 174/500:  35%|███▍      | 174/500 [19:58<37:35,  6.92s/it, loss=1.48e+04, v_num=1]
```

Example 583 (unknown):
```unknown
Epoch 174/500:  35%|███▍      | 174/500 [19:58<37:35,  6.92s/it, loss=1.5e+04, v_num=1]
```

Example 584 (unknown):
```unknown
Epoch 175/500:  35%|███▍      | 174/500 [19:58<37:35,  6.92s/it, loss=1.5e+04, v_num=1]
```

Example 585 (unknown):
```unknown
Epoch 175/500:  35%|███▌      | 175/500 [20:05<37:16,  6.88s/it, loss=1.5e+04, v_num=1]
```

Example 586 (unknown):
```unknown
Epoch 175/500:  35%|███▌      | 175/500 [20:05<37:16,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 587 (unknown):
```unknown
Epoch 176/500:  35%|███▌      | 175/500 [20:05<37:16,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 588 (unknown):
```unknown
Epoch 176/500:  35%|███▌      | 176/500 [20:12<37:15,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 589 (unknown):
```unknown
Epoch 176/500:  35%|███▌      | 176/500 [20:12<37:15,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 590 (unknown):
```unknown
Epoch 177/500:  35%|███▌      | 176/500 [20:12<37:15,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 591 (unknown):
```unknown
Epoch 177/500:  35%|███▌      | 177/500 [20:19<37:10,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 592 (unknown):
```unknown
Epoch 177/500:  35%|███▌      | 177/500 [20:19<37:10,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 593 (unknown):
```unknown
Epoch 178/500:  35%|███▌      | 177/500 [20:19<37:10,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 594 (unknown):
```unknown
Epoch 178/500:  36%|███▌      | 178/500 [20:25<36:45,  6.85s/it, loss=1.52e+04, v_num=1]
```

Example 595 (unknown):
```unknown
Epoch 178/500:  36%|███▌      | 178/500 [20:25<36:45,  6.85s/it, loss=1.51e+04, v_num=1]
```

Example 596 (unknown):
```unknown
Epoch 179/500:  36%|███▌      | 178/500 [20:26<36:45,  6.85s/it, loss=1.51e+04, v_num=1]
```

Example 597 (unknown):
```unknown
Epoch 179/500:  36%|███▌      | 179/500 [20:32<36:50,  6.89s/it, loss=1.51e+04, v_num=1]
```

Example 598 (unknown):
```unknown
Epoch 179/500:  36%|███▌      | 179/500 [20:32<36:50,  6.89s/it, loss=1.51e+04, v_num=1]
```

Example 599 (unknown):
```unknown
Epoch 180/500:  36%|███▌      | 179/500 [20:32<36:50,  6.89s/it, loss=1.51e+04, v_num=1]
```

Example 600 (unknown):
```unknown
Epoch 180/500:  36%|███▌      | 180/500 [20:39<36:44,  6.89s/it, loss=1.51e+04, v_num=1]
```

Example 601 (unknown):
```unknown
Epoch 180/500:  36%|███▌      | 180/500 [20:39<36:44,  6.89s/it, loss=1.48e+04, v_num=1]
```

Example 602 (unknown):
```unknown
Epoch 181/500:  36%|███▌      | 180/500 [20:39<36:44,  6.89s/it, loss=1.48e+04, v_num=1]
```

Example 603 (unknown):
```unknown
Epoch 181/500:  36%|███▌      | 181/500 [20:46<36:25,  6.85s/it, loss=1.48e+04, v_num=1]
```

Example 604 (unknown):
```unknown
Epoch 181/500:  36%|███▌      | 181/500 [20:46<36:25,  6.85s/it, loss=1.51e+04, v_num=1]
```

Example 605 (unknown):
```unknown
Epoch 182/500:  36%|███▌      | 181/500 [20:46<36:25,  6.85s/it, loss=1.51e+04, v_num=1]
```

Example 606 (unknown):
```unknown
Epoch 182/500:  36%|███▋      | 182/500 [20:53<36:26,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 607 (unknown):
```unknown
Epoch 182/500:  36%|███▋      | 182/500 [20:53<36:26,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 608 (unknown):
```unknown
Epoch 183/500:  36%|███▋      | 182/500 [20:53<36:26,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 609 (unknown):
```unknown
Epoch 183/500:  37%|███▋      | 183/500 [21:00<36:26,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 610 (unknown):
```unknown
Epoch 183/500:  37%|███▋      | 183/500 [21:00<36:26,  6.90s/it, loss=1.48e+04, v_num=1]
```

Example 611 (unknown):
```unknown
Epoch 184/500:  37%|███▋      | 183/500 [21:00<36:26,  6.90s/it, loss=1.48e+04, v_num=1]
```

Example 612 (unknown):
```unknown
Epoch 184/500:  37%|███▋      | 184/500 [21:07<36:15,  6.88s/it, loss=1.48e+04, v_num=1]
```

Example 613 (unknown):
```unknown
Epoch 184/500:  37%|███▋      | 184/500 [21:07<36:15,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 614 (unknown):
```unknown
Epoch 185/500:  37%|███▋      | 184/500 [21:07<36:15,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 615 (unknown):
```unknown
Epoch 185/500:  37%|███▋      | 185/500 [21:14<36:02,  6.86s/it, loss=1.51e+04, v_num=1]
```

Example 616 (unknown):
```unknown
Epoch 185/500:  37%|███▋      | 185/500 [21:14<36:02,  6.86s/it, loss=1.51e+04, v_num=1]
```

Example 617 (unknown):
```unknown
Epoch 186/500:  37%|███▋      | 185/500 [21:14<36:02,  6.86s/it, loss=1.51e+04, v_num=1]
```

Example 618 (unknown):
```unknown
Epoch 186/500:  37%|███▋      | 186/500 [21:21<36:02,  6.89s/it, loss=1.51e+04, v_num=1]
```

Example 619 (unknown):
```unknown
Epoch 186/500:  37%|███▋      | 186/500 [21:21<36:02,  6.89s/it, loss=1.5e+04, v_num=1]
```

Example 620 (unknown):
```unknown
Epoch 187/500:  37%|███▋      | 186/500 [21:21<36:02,  6.89s/it, loss=1.5e+04, v_num=1]
```

Example 621 (unknown):
```unknown
Epoch 187/500:  37%|███▋      | 187/500 [21:28<35:59,  6.90s/it, loss=1.5e+04, v_num=1]
```

Example 622 (unknown):
```unknown
Epoch 187/500:  37%|███▋      | 187/500 [21:28<35:59,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 623 (unknown):
```unknown
Epoch 188/500:  37%|███▋      | 187/500 [21:28<35:59,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 624 (unknown):
```unknown
Epoch 188/500:  38%|███▊      | 188/500 [21:35<36:03,  6.93s/it, loss=1.52e+04, v_num=1]
```

Example 625 (unknown):
```unknown
Epoch 188/500:  38%|███▊      | 188/500 [21:35<36:03,  6.93s/it, loss=1.52e+04, v_num=1]
```

Example 626 (unknown):
```unknown
Epoch 189/500:  38%|███▊      | 188/500 [21:35<36:03,  6.93s/it, loss=1.52e+04, v_num=1]
```

Example 627 (unknown):
```unknown
Epoch 189/500:  38%|███▊      | 189/500 [21:42<35:58,  6.94s/it, loss=1.52e+04, v_num=1]
```

Example 628 (unknown):
```unknown
Epoch 189/500:  38%|███▊      | 189/500 [21:42<35:58,  6.94s/it, loss=1.54e+04, v_num=1]
```

Example 629 (unknown):
```unknown
Epoch 190/500:  38%|███▊      | 189/500 [21:42<35:58,  6.94s/it, loss=1.54e+04, v_num=1]
```

Example 630 (unknown):
```unknown
Epoch 190/500:  38%|███▊      | 190/500 [21:48<35:16,  6.83s/it, loss=1.54e+04, v_num=1]
```

Example 631 (unknown):
```unknown
Epoch 190/500:  38%|███▊      | 190/500 [21:48<35:16,  6.83s/it, loss=1.53e+04, v_num=1]
```

Example 632 (unknown):
```unknown
Epoch 191/500:  38%|███▊      | 190/500 [21:48<35:16,  6.83s/it, loss=1.53e+04, v_num=1]
```

Example 633 (unknown):
```unknown
Epoch 191/500:  38%|███▊      | 191/500 [21:55<35:20,  6.86s/it, loss=1.53e+04, v_num=1]
```

Example 634 (unknown):
```unknown
Epoch 191/500:  38%|███▊      | 191/500 [21:55<35:20,  6.86s/it, loss=1.52e+04, v_num=1]
```

Example 635 (unknown):
```unknown
Epoch 192/500:  38%|███▊      | 191/500 [21:55<35:20,  6.86s/it, loss=1.52e+04, v_num=1]
```

Example 636 (unknown):
```unknown
Epoch 192/500:  38%|███▊      | 192/500 [22:02<35:22,  6.89s/it, loss=1.52e+04, v_num=1]
```

Example 637 (unknown):
```unknown
Epoch 192/500:  38%|███▊      | 192/500 [22:02<35:22,  6.89s/it, loss=1.5e+04, v_num=1]
```

Example 638 (unknown):
```unknown
Epoch 193/500:  38%|███▊      | 192/500 [22:02<35:22,  6.89s/it, loss=1.5e+04, v_num=1]
```

Example 639 (unknown):
```unknown
Epoch 193/500:  39%|███▊      | 193/500 [22:09<34:49,  6.81s/it, loss=1.5e+04, v_num=1]
```

Example 640 (unknown):
```unknown
Epoch 193/500:  39%|███▊      | 193/500 [22:09<34:49,  6.81s/it, loss=1.51e+04, v_num=1]
```

Example 641 (unknown):
```unknown
Epoch 194/500:  39%|███▊      | 193/500 [22:09<34:49,  6.81s/it, loss=1.51e+04, v_num=1]
```

Example 642 (unknown):
```unknown
Epoch 194/500:  39%|███▉      | 194/500 [22:15<34:50,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 643 (unknown):
```unknown
Epoch 194/500:  39%|███▉      | 194/500 [22:15<34:50,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 644 (unknown):
```unknown
Epoch 195/500:  39%|███▉      | 194/500 [22:15<34:50,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 645 (unknown):
```unknown
Epoch 195/500:  39%|███▉      | 195/500 [22:23<35:03,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 646 (unknown):
```unknown
Epoch 195/500:  39%|███▉      | 195/500 [22:23<35:03,  6.90s/it, loss=1.5e+04, v_num=1]
```

Example 647 (unknown):
```unknown
Epoch 196/500:  39%|███▉      | 195/500 [22:23<35:03,  6.90s/it, loss=1.5e+04, v_num=1]
```

Example 648 (unknown):
```unknown
Epoch 196/500:  39%|███▉      | 196/500 [22:30<35:09,  6.94s/it, loss=1.5e+04, v_num=1]
```

Example 649 (unknown):
```unknown
Epoch 196/500:  39%|███▉      | 196/500 [22:30<35:09,  6.94s/it, loss=1.52e+04, v_num=1]
```

Example 650 (unknown):
```unknown
Epoch 197/500:  39%|███▉      | 196/500 [22:30<35:09,  6.94s/it, loss=1.52e+04, v_num=1]
```

Example 651 (unknown):
```unknown
Epoch 197/500:  39%|███▉      | 197/500 [22:36<34:43,  6.88s/it, loss=1.52e+04, v_num=1]
```

Example 652 (unknown):
```unknown
Epoch 197/500:  39%|███▉      | 197/500 [22:36<34:43,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 653 (unknown):
```unknown
Epoch 198/500:  39%|███▉      | 197/500 [22:36<34:43,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 654 (unknown):
```unknown
Epoch 198/500:  40%|███▉      | 198/500 [22:43<34:31,  6.86s/it, loss=1.51e+04, v_num=1]
```

Example 655 (unknown):
```unknown
Epoch 198/500:  40%|███▉      | 198/500 [22:43<34:31,  6.86s/it, loss=1.51e+04, v_num=1]
```

Example 656 (unknown):
```unknown
Epoch 199/500:  40%|███▉      | 198/500 [22:43<34:31,  6.86s/it, loss=1.51e+04, v_num=1]
```

Example 657 (unknown):
```unknown
Epoch 199/500:  40%|███▉      | 199/500 [22:50<33:58,  6.77s/it, loss=1.51e+04, v_num=1]
```

Example 658 (unknown):
```unknown
Epoch 199/500:  40%|███▉      | 199/500 [22:50<33:58,  6.77s/it, loss=1.49e+04, v_num=1]
```

Example 659 (unknown):
```unknown
Epoch 200/500:  40%|███▉      | 199/500 [22:50<33:58,  6.77s/it, loss=1.49e+04, v_num=1]
```

Example 660 (unknown):
```unknown
Epoch 200/500:  40%|████      | 200/500 [22:57<34:01,  6.80s/it, loss=1.49e+04, v_num=1]
```

Example 661 (unknown):
```unknown
Epoch 200/500:  40%|████      | 200/500 [22:57<34:01,  6.80s/it, loss=1.5e+04, v_num=1]
```

Example 662 (unknown):
```unknown
Epoch 201/500:  40%|████      | 200/500 [22:57<34:01,  6.80s/it, loss=1.5e+04, v_num=1]
```

Example 663 (unknown):
```unknown
Epoch 201/500:  40%|████      | 201/500 [23:04<34:05,  6.84s/it, loss=1.5e+04, v_num=1]
```

Example 664 (unknown):
```unknown
Epoch 201/500:  40%|████      | 201/500 [23:04<34:05,  6.84s/it, loss=1.5e+04, v_num=1]
```

Example 665 (unknown):
```unknown
Epoch 202/500:  40%|████      | 201/500 [23:04<34:05,  6.84s/it, loss=1.5e+04, v_num=1]
```

Example 666 (unknown):
```unknown
Epoch 202/500:  40%|████      | 202/500 [23:10<34:06,  6.87s/it, loss=1.5e+04, v_num=1]
```

Example 667 (unknown):
```unknown
Epoch 202/500:  40%|████      | 202/500 [23:10<34:06,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 668 (unknown):
```unknown
Epoch 203/500:  40%|████      | 202/500 [23:10<34:06,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 669 (unknown):
```unknown
Epoch 203/500:  41%|████      | 203/500 [23:17<34:08,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 670 (unknown):
```unknown
Epoch 203/500:  41%|████      | 203/500 [23:17<34:08,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 671 (unknown):
```unknown
Epoch 204/500:  41%|████      | 203/500 [23:17<34:08,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 672 (unknown):
```unknown
Epoch 204/500:  41%|████      | 204/500 [23:24<33:53,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 673 (unknown):
```unknown
Epoch 204/500:  41%|████      | 204/500 [23:24<33:53,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 674 (unknown):
```unknown
Epoch 205/500:  41%|████      | 204/500 [23:24<33:53,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 675 (unknown):
```unknown
Epoch 205/500:  41%|████      | 205/500 [23:31<33:23,  6.79s/it, loss=1.51e+04, v_num=1]
```

Example 676 (unknown):
```unknown
Epoch 205/500:  41%|████      | 205/500 [23:31<33:23,  6.79s/it, loss=1.54e+04, v_num=1]
```

Example 677 (unknown):
```unknown
Epoch 206/500:  41%|████      | 205/500 [23:31<33:23,  6.79s/it, loss=1.54e+04, v_num=1]
```

Example 678 (unknown):
```unknown
Epoch 206/500:  41%|████      | 206/500 [23:38<33:26,  6.83s/it, loss=1.54e+04, v_num=1]
```

Example 679 (unknown):
```unknown
Epoch 206/500:  41%|████      | 206/500 [23:38<33:26,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 680 (unknown):
```unknown
Epoch 207/500:  41%|████      | 206/500 [23:38<33:26,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 681 (unknown):
```unknown
Epoch 207/500:  41%|████▏     | 207/500 [23:45<33:35,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 682 (unknown):
```unknown
Epoch 207/500:  41%|████▏     | 207/500 [23:45<33:35,  6.88s/it, loss=1.52e+04, v_num=1]
```

Example 683 (unknown):
```unknown
Epoch 208/500:  41%|████▏     | 207/500 [23:45<33:35,  6.88s/it, loss=1.52e+04, v_num=1]
```

Example 684 (unknown):
```unknown
Epoch 208/500:  42%|████▏     | 208/500 [23:52<33:38,  6.91s/it, loss=1.52e+04, v_num=1]
```

Example 685 (unknown):
```unknown
Epoch 208/500:  42%|████▏     | 208/500 [23:52<33:38,  6.91s/it, loss=1.49e+04, v_num=1]
```

Example 686 (unknown):
```unknown
Epoch 209/500:  42%|████▏     | 208/500 [23:52<33:38,  6.91s/it, loss=1.49e+04, v_num=1]
```

Example 687 (unknown):
```unknown
Epoch 209/500:  42%|████▏     | 209/500 [23:58<33:08,  6.83s/it, loss=1.49e+04, v_num=1]
```

Example 688 (unknown):
```unknown
Epoch 209/500:  42%|████▏     | 209/500 [23:58<33:08,  6.83s/it, loss=1.49e+04, v_num=1]
```

Example 689 (unknown):
```unknown
Epoch 210/500:  42%|████▏     | 209/500 [23:58<33:08,  6.83s/it, loss=1.49e+04, v_num=1]
```

Example 690 (unknown):
```unknown
Epoch 210/500:  42%|████▏     | 210/500 [24:05<33:00,  6.83s/it, loss=1.49e+04, v_num=1]
```

Example 691 (unknown):
```unknown
Epoch 210/500:  42%|████▏     | 210/500 [24:05<33:00,  6.83s/it, loss=1.52e+04, v_num=1]
```

Example 692 (unknown):
```unknown
Epoch 211/500:  42%|████▏     | 210/500 [24:05<33:00,  6.83s/it, loss=1.52e+04, v_num=1]
```

Example 693 (unknown):
```unknown
Epoch 211/500:  42%|████▏     | 211/500 [24:12<32:40,  6.78s/it, loss=1.52e+04, v_num=1]
```

Example 694 (unknown):
```unknown
Epoch 211/500:  42%|████▏     | 211/500 [24:12<32:40,  6.78s/it, loss=1.49e+04, v_num=1]
```

Example 695 (unknown):
```unknown
Epoch 212/500:  42%|████▏     | 211/500 [24:12<32:40,  6.78s/it, loss=1.49e+04, v_num=1]
```

Example 696 (unknown):
```unknown
Epoch 212/500:  42%|████▏     | 212/500 [24:19<32:50,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 697 (unknown):
```unknown
Epoch 212/500:  42%|████▏     | 212/500 [24:19<32:50,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 698 (unknown):
```unknown
Epoch 213/500:  42%|████▏     | 212/500 [24:19<32:50,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 699 (unknown):
```unknown
Epoch 213/500:  43%|████▎     | 213/500 [24:26<32:39,  6.83s/it, loss=1.52e+04, v_num=1]
```

Example 700 (unknown):
```unknown
Epoch 213/500:  43%|████▎     | 213/500 [24:26<32:39,  6.83s/it, loss=1.49e+04, v_num=1]
```

Example 701 (unknown):
```unknown
Epoch 214/500:  43%|████▎     | 213/500 [24:26<32:39,  6.83s/it, loss=1.49e+04, v_num=1]
```

Example 702 (unknown):
```unknown
Epoch 214/500:  43%|████▎     | 214/500 [24:32<32:24,  6.80s/it, loss=1.49e+04, v_num=1]
```

Example 703 (unknown):
```unknown
Epoch 214/500:  43%|████▎     | 214/500 [24:32<32:24,  6.80s/it, loss=1.49e+04, v_num=1]
```

Example 704 (unknown):
```unknown
Epoch 215/500:  43%|████▎     | 214/500 [24:32<32:24,  6.80s/it, loss=1.49e+04, v_num=1]
```

Example 705 (unknown):
```unknown
Epoch 215/500:  43%|████▎     | 215/500 [24:39<32:23,  6.82s/it, loss=1.49e+04, v_num=1]
```

Example 706 (unknown):
```unknown
Epoch 215/500:  43%|████▎     | 215/500 [24:39<32:23,  6.82s/it, loss=1.52e+04, v_num=1]
```

Example 707 (unknown):
```unknown
Epoch 216/500:  43%|████▎     | 215/500 [24:39<32:23,  6.82s/it, loss=1.52e+04, v_num=1]
```

Example 708 (unknown):
```unknown
Epoch 216/500:  43%|████▎     | 216/500 [24:46<32:32,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 709 (unknown):
```unknown
Epoch 216/500:  43%|████▎     | 216/500 [24:46<32:32,  6.87s/it, loss=1.53e+04, v_num=1]
```

Example 710 (unknown):
```unknown
Epoch 217/500:  43%|████▎     | 216/500 [24:46<32:32,  6.87s/it, loss=1.53e+04, v_num=1]
```

Example 711 (unknown):
```unknown
Epoch 217/500:  43%|████▎     | 217/500 [24:53<32:23,  6.87s/it, loss=1.53e+04, v_num=1]
```

Example 712 (unknown):
```unknown
Epoch 217/500:  43%|████▎     | 217/500 [24:53<32:23,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 713 (unknown):
```unknown
Epoch 218/500:  43%|████▎     | 217/500 [24:53<32:23,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 714 (unknown):
```unknown
Epoch 218/500:  44%|████▎     | 218/500 [25:00<32:29,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 715 (unknown):
```unknown
Epoch 218/500:  44%|████▎     | 218/500 [25:00<32:29,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 716 (unknown):
```unknown
Epoch 219/500:  44%|████▎     | 218/500 [25:00<32:29,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 717 (unknown):
```unknown
Epoch 219/500:  44%|████▍     | 219/500 [25:07<32:05,  6.85s/it, loss=1.5e+04, v_num=1]
```

Example 718 (unknown):
```unknown
Epoch 219/500:  44%|████▍     | 219/500 [25:07<32:05,  6.85s/it, loss=1.56e+04, v_num=1]
```

Example 719 (unknown):
```unknown
Epoch 220/500:  44%|████▍     | 219/500 [25:07<32:05,  6.85s/it, loss=1.56e+04, v_num=1]
```

Example 720 (unknown):
```unknown
Epoch 220/500:  44%|████▍     | 220/500 [25:14<32:04,  6.87s/it, loss=1.56e+04, v_num=1]
```

Example 721 (unknown):
```unknown
Epoch 220/500:  44%|████▍     | 220/500 [25:14<32:04,  6.87s/it, loss=1.5e+04, v_num=1]
```

Example 722 (unknown):
```unknown
Epoch 221/500:  44%|████▍     | 220/500 [25:14<32:04,  6.87s/it, loss=1.5e+04, v_num=1]
```

Example 723 (unknown):
```unknown
Epoch 221/500:  44%|████▍     | 221/500 [25:21<31:56,  6.87s/it, loss=1.5e+04, v_num=1]
```

Example 724 (unknown):
```unknown
Epoch 221/500:  44%|████▍     | 221/500 [25:21<31:56,  6.87s/it, loss=1.5e+04, v_num=1]
```

Example 725 (unknown):
```unknown
Epoch 222/500:  44%|████▍     | 221/500 [25:21<31:56,  6.87s/it, loss=1.5e+04, v_num=1]
```

Example 726 (unknown):
```unknown
Epoch 222/500:  44%|████▍     | 222/500 [25:28<31:53,  6.88s/it, loss=1.5e+04, v_num=1]
```

Example 727 (unknown):
```unknown
Epoch 222/500:  44%|████▍     | 222/500 [25:28<31:53,  6.88s/it, loss=1.55e+04, v_num=1]
```

Example 728 (unknown):
```unknown
Epoch 223/500:  44%|████▍     | 222/500 [25:28<31:53,  6.88s/it, loss=1.55e+04, v_num=1]
```

Example 729 (unknown):
```unknown
Epoch 223/500:  45%|████▍     | 223/500 [25:35<31:55,  6.92s/it, loss=1.55e+04, v_num=1]
```

Example 730 (unknown):
```unknown
Epoch 223/500:  45%|████▍     | 223/500 [25:35<31:55,  6.92s/it, loss=1.49e+04, v_num=1]
```

Example 731 (unknown):
```unknown
Epoch 224/500:  45%|████▍     | 223/500 [25:35<31:55,  6.92s/it, loss=1.49e+04, v_num=1]
```

Example 732 (unknown):
```unknown
Epoch 224/500:  45%|████▍     | 224/500 [25:42<32:02,  6.97s/it, loss=1.49e+04, v_num=1]
```

Example 733 (unknown):
```unknown
Epoch 224/500:  45%|████▍     | 224/500 [25:42<32:02,  6.97s/it, loss=1.5e+04, v_num=1]
```

Example 734 (unknown):
```unknown
Epoch 225/500:  45%|████▍     | 224/500 [25:42<32:02,  6.97s/it, loss=1.5e+04, v_num=1]
```

Example 735 (unknown):
```unknown
Epoch 225/500:  45%|████▌     | 225/500 [25:49<31:56,  6.97s/it, loss=1.5e+04, v_num=1]
```

Example 736 (unknown):
```unknown
Epoch 225/500:  45%|████▌     | 225/500 [25:49<31:56,  6.97s/it, loss=1.51e+04, v_num=1]
```

Example 737 (unknown):
```unknown
Epoch 226/500:  45%|████▌     | 225/500 [25:49<31:56,  6.97s/it, loss=1.51e+04, v_num=1]
```

Example 738 (unknown):
```unknown
Epoch 226/500:  45%|████▌     | 226/500 [25:56<31:49,  6.97s/it, loss=1.51e+04, v_num=1]
```

Example 739 (unknown):
```unknown
Epoch 226/500:  45%|████▌     | 226/500 [25:56<31:49,  6.97s/it, loss=1.47e+04, v_num=1]
```

Example 740 (unknown):
```unknown
Epoch 227/500:  45%|████▌     | 226/500 [25:56<31:49,  6.97s/it, loss=1.47e+04, v_num=1]
```

Example 741 (unknown):
```unknown
Epoch 227/500:  45%|████▌     | 227/500 [26:02<31:40,  6.96s/it, loss=1.47e+04, v_num=1]
```

Example 742 (unknown):
```unknown
Epoch 227/500:  45%|████▌     | 227/500 [26:02<31:40,  6.96s/it, loss=1.51e+04, v_num=1]
```

Example 743 (unknown):
```unknown
Epoch 228/500:  45%|████▌     | 227/500 [26:02<31:40,  6.96s/it, loss=1.51e+04, v_num=1]
```

Example 744 (unknown):
```unknown
Epoch 228/500:  46%|████▌     | 228/500 [26:09<31:01,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 745 (unknown):
```unknown
Epoch 228/500:  46%|████▌     | 228/500 [26:09<31:01,  6.84s/it, loss=1.5e+04, v_num=1]
```

Example 746 (unknown):
```unknown
Epoch 229/500:  46%|████▌     | 228/500 [26:09<31:01,  6.84s/it, loss=1.5e+04, v_num=1]
```

Example 747 (unknown):
```unknown
Epoch 229/500:  46%|████▌     | 229/500 [26:16<30:52,  6.84s/it, loss=1.5e+04, v_num=1]
```

Example 748 (unknown):
```unknown
Epoch 229/500:  46%|████▌     | 229/500 [26:16<30:52,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 749 (unknown):
```unknown
Epoch 230/500:  46%|████▌     | 229/500 [26:16<30:52,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 750 (unknown):
```unknown
Epoch 230/500:  46%|████▌     | 230/500 [26:23<30:37,  6.81s/it, loss=1.52e+04, v_num=1]
```

Example 751 (unknown):
```unknown
Epoch 230/500:  46%|████▌     | 230/500 [26:23<30:37,  6.81s/it, loss=1.52e+04, v_num=1]
```

Example 752 (unknown):
```unknown
Epoch 231/500:  46%|████▌     | 230/500 [26:23<30:37,  6.81s/it, loss=1.52e+04, v_num=1]
```

Example 753 (unknown):
```unknown
Epoch 231/500:  46%|████▌     | 231/500 [26:30<30:39,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 754 (unknown):
```unknown
Epoch 231/500:  46%|████▌     | 231/500 [26:30<30:39,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 755 (unknown):
```unknown
Epoch 232/500:  46%|████▌     | 231/500 [26:30<30:39,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 756 (unknown):
```unknown
Epoch 232/500:  46%|████▋     | 232/500 [26:36<30:43,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 757 (unknown):
```unknown
Epoch 232/500:  46%|████▋     | 232/500 [26:36<30:43,  6.88s/it, loss=1.54e+04, v_num=1]
```

Example 758 (unknown):
```unknown
Epoch 233/500:  46%|████▋     | 232/500 [26:36<30:43,  6.88s/it, loss=1.54e+04, v_num=1]
```

Example 759 (unknown):
```unknown
Epoch 233/500:  47%|████▋     | 233/500 [26:43<30:45,  6.91s/it, loss=1.54e+04, v_num=1]
```

Example 760 (unknown):
```unknown
Epoch 233/500:  47%|████▋     | 233/500 [26:43<30:45,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 761 (unknown):
```unknown
Epoch 234/500:  47%|████▋     | 233/500 [26:43<30:45,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 762 (unknown):
```unknown
Epoch 234/500:  47%|████▋     | 234/500 [26:50<30:41,  6.92s/it, loss=1.5e+04, v_num=1]
```

Example 763 (unknown):
```unknown
Epoch 234/500:  47%|████▋     | 234/500 [26:50<30:41,  6.92s/it, loss=1.49e+04, v_num=1]
```

Example 764 (unknown):
```unknown
Epoch 235/500:  47%|████▋     | 234/500 [26:50<30:41,  6.92s/it, loss=1.49e+04, v_num=1]
```

Example 765 (unknown):
```unknown
Epoch 235/500:  47%|████▋     | 235/500 [26:57<30:34,  6.92s/it, loss=1.49e+04, v_num=1]
```

Example 766 (unknown):
```unknown
Epoch 235/500:  47%|████▋     | 235/500 [26:57<30:34,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 767 (unknown):
```unknown
Epoch 236/500:  47%|████▋     | 235/500 [26:57<30:34,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 768 (unknown):
```unknown
Epoch 236/500:  47%|████▋     | 236/500 [27:04<30:31,  6.94s/it, loss=1.51e+04, v_num=1]
```

Example 769 (unknown):
```unknown
Epoch 236/500:  47%|████▋     | 236/500 [27:04<30:31,  6.94s/it, loss=1.52e+04, v_num=1]
```

Example 770 (unknown):
```unknown
Epoch 237/500:  47%|████▋     | 236/500 [27:04<30:31,  6.94s/it, loss=1.52e+04, v_num=1]
```

Example 771 (unknown):
```unknown
Epoch 237/500:  47%|████▋     | 237/500 [27:11<30:31,  6.97s/it, loss=1.52e+04, v_num=1]
```

Example 772 (unknown):
```unknown
Epoch 237/500:  47%|████▋     | 237/500 [27:11<30:31,  6.97s/it, loss=1.48e+04, v_num=1]
```

Example 773 (unknown):
```unknown
Epoch 238/500:  47%|████▋     | 237/500 [27:11<30:31,  6.97s/it, loss=1.48e+04, v_num=1]
```

Example 774 (unknown):
```unknown
Epoch 238/500:  48%|████▊     | 238/500 [27:18<30:23,  6.96s/it, loss=1.48e+04, v_num=1]
```

Example 775 (unknown):
```unknown
Epoch 238/500:  48%|████▊     | 238/500 [27:18<30:23,  6.96s/it, loss=1.51e+04, v_num=1]
```

Example 776 (unknown):
```unknown
Epoch 239/500:  48%|████▊     | 238/500 [27:18<30:23,  6.96s/it, loss=1.51e+04, v_num=1]
```

Example 777 (unknown):
```unknown
Epoch 239/500:  48%|████▊     | 239/500 [27:25<30:11,  6.94s/it, loss=1.51e+04, v_num=1]
```

Example 778 (unknown):
```unknown
Epoch 239/500:  48%|████▊     | 239/500 [27:25<30:11,  6.94s/it, loss=1.49e+04, v_num=1]
```

Example 779 (unknown):
```unknown
Epoch 240/500:  48%|████▊     | 239/500 [27:25<30:11,  6.94s/it, loss=1.49e+04, v_num=1]
```

Example 780 (unknown):
```unknown
Epoch 240/500:  48%|████▊     | 240/500 [27:32<30:06,  6.95s/it, loss=1.49e+04, v_num=1]
```

Example 781 (unknown):
```unknown
Epoch 240/500:  48%|████▊     | 240/500 [27:32<30:06,  6.95s/it, loss=1.51e+04, v_num=1]
```

Example 782 (unknown):
```unknown
Epoch 241/500:  48%|████▊     | 240/500 [27:32<30:06,  6.95s/it, loss=1.51e+04, v_num=1]
```

Example 783 (unknown):
```unknown
Epoch 241/500:  48%|████▊     | 241/500 [27:39<30:03,  6.96s/it, loss=1.51e+04, v_num=1]
```

Example 784 (unknown):
```unknown
Epoch 241/500:  48%|████▊     | 241/500 [27:39<30:03,  6.96s/it, loss=1.5e+04, v_num=1]
```

Example 785 (unknown):
```unknown
Epoch 242/500:  48%|████▊     | 241/500 [27:39<30:03,  6.96s/it, loss=1.5e+04, v_num=1]
```

Example 786 (unknown):
```unknown
Epoch 242/500:  48%|████▊     | 242/500 [27:46<29:46,  6.93s/it, loss=1.5e+04, v_num=1]
```

Example 787 (unknown):
```unknown
Epoch 242/500:  48%|████▊     | 242/500 [27:46<29:46,  6.93s/it, loss=1.53e+04, v_num=1]
```

Example 788 (unknown):
```unknown
Epoch 243/500:  48%|████▊     | 242/500 [27:46<29:46,  6.93s/it, loss=1.53e+04, v_num=1]
```

Example 789 (unknown):
```unknown
Epoch 243/500:  49%|████▊     | 243/500 [27:53<29:37,  6.92s/it, loss=1.53e+04, v_num=1]
```

Example 790 (unknown):
```unknown
Epoch 243/500:  49%|████▊     | 243/500 [27:53<29:37,  6.92s/it, loss=1.54e+04, v_num=1]
```

Example 791 (unknown):
```unknown
Epoch 244/500:  49%|████▊     | 243/500 [27:53<29:37,  6.92s/it, loss=1.54e+04, v_num=1]
```

Example 792 (unknown):
```unknown
Epoch 244/500:  49%|████▉     | 244/500 [28:00<29:11,  6.84s/it, loss=1.54e+04, v_num=1]
```

Example 793 (unknown):
```unknown
Epoch 244/500:  49%|████▉     | 244/500 [28:00<29:11,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 794 (unknown):
```unknown
Epoch 245/500:  49%|████▉     | 244/500 [28:00<29:11,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 795 (unknown):
```unknown
Epoch 245/500:  49%|████▉     | 245/500 [28:07<29:12,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 796 (unknown):
```unknown
Epoch 245/500:  49%|████▉     | 245/500 [28:07<29:12,  6.87s/it, loss=1.49e+04, v_num=1]
```

Example 797 (unknown):
```unknown
Epoch 246/500:  49%|████▉     | 245/500 [28:07<29:12,  6.87s/it, loss=1.49e+04, v_num=1]
```

Example 798 (unknown):
```unknown
Epoch 246/500:  49%|████▉     | 246/500 [28:13<29:05,  6.87s/it, loss=1.49e+04, v_num=1]
```

Example 799 (unknown):
```unknown
Epoch 246/500:  49%|████▉     | 246/500 [28:13<29:05,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 800 (unknown):
```unknown
Epoch 247/500:  49%|████▉     | 246/500 [28:13<29:05,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 801 (unknown):
```unknown
Epoch 247/500:  49%|████▉     | 247/500 [28:20<29:06,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 802 (unknown):
```unknown
Epoch 247/500:  49%|████▉     | 247/500 [28:20<29:06,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 803 (unknown):
```unknown
Epoch 248/500:  49%|████▉     | 247/500 [28:20<29:06,  6.90s/it, loss=1.52e+04, v_num=1]
```

Example 804 (unknown):
```unknown
Epoch 248/500:  50%|████▉     | 248/500 [28:27<28:54,  6.88s/it, loss=1.52e+04, v_num=1]
```

Example 805 (unknown):
```unknown
Epoch 248/500:  50%|████▉     | 248/500 [28:27<28:54,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 806 (unknown):
```unknown
Epoch 249/500:  50%|████▉     | 248/500 [28:27<28:54,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 807 (unknown):
```unknown
Epoch 249/500:  50%|████▉     | 249/500 [28:34<28:36,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 808 (unknown):
```unknown
Epoch 249/500:  50%|████▉     | 249/500 [28:34<28:36,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 809 (unknown):
```unknown
Epoch 250/500:  50%|████▉     | 249/500 [28:34<28:36,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 810 (unknown):
```unknown
Epoch 250/500:  50%|█████     | 250/500 [28:41<28:14,  6.78s/it, loss=1.49e+04, v_num=1]
```

Example 811 (unknown):
```unknown
Epoch 250/500:  50%|█████     | 250/500 [28:41<28:14,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 812 (unknown):
```unknown
Epoch 251/500:  50%|█████     | 250/500 [28:41<28:14,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 813 (unknown):
```unknown
Epoch 251/500:  50%|█████     | 251/500 [28:48<28:21,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 814 (unknown):
```unknown
Epoch 251/500:  50%|█████     | 251/500 [28:48<28:21,  6.83s/it, loss=1.53e+04, v_num=1]
```

Example 815 (unknown):
```unknown
Epoch 252/500:  50%|█████     | 251/500 [28:48<28:21,  6.83s/it, loss=1.53e+04, v_num=1]
```

Example 816 (unknown):
```unknown
Epoch 252/500:  50%|█████     | 252/500 [28:54<28:06,  6.80s/it, loss=1.53e+04, v_num=1]
```

Example 817 (unknown):
```unknown
Epoch 252/500:  50%|█████     | 252/500 [28:54<28:06,  6.80s/it, loss=1.48e+04, v_num=1]
```

Example 818 (unknown):
```unknown
Epoch 253/500:  50%|█████     | 252/500 [28:54<28:06,  6.80s/it, loss=1.48e+04, v_num=1]
```

Example 819 (unknown):
```unknown
Epoch 253/500:  51%|█████     | 253/500 [29:01<28:07,  6.83s/it, loss=1.48e+04, v_num=1]
```

Example 820 (unknown):
```unknown
Epoch 253/500:  51%|█████     | 253/500 [29:01<28:07,  6.83s/it, loss=1.54e+04, v_num=1]
```

Example 821 (unknown):
```unknown
Epoch 254/500:  51%|█████     | 253/500 [29:01<28:07,  6.83s/it, loss=1.54e+04, v_num=1]
```

Example 822 (unknown):
```unknown
Epoch 254/500:  51%|█████     | 254/500 [29:08<28:00,  6.83s/it, loss=1.54e+04, v_num=1]
```

Example 823 (unknown):
```unknown
Epoch 254/500:  51%|█████     | 254/500 [29:08<28:00,  6.83s/it, loss=1.47e+04, v_num=1]
```

Example 824 (unknown):
```unknown
Epoch 255/500:  51%|█████     | 254/500 [29:08<28:00,  6.83s/it, loss=1.47e+04, v_num=1]
```

Example 825 (unknown):
```unknown
Epoch 255/500:  51%|█████     | 255/500 [29:15<27:48,  6.81s/it, loss=1.47e+04, v_num=1]
```

Example 826 (unknown):
```unknown
Epoch 255/500:  51%|█████     | 255/500 [29:15<27:48,  6.81s/it, loss=1.49e+04, v_num=1]
```

Example 827 (unknown):
```unknown
Epoch 256/500:  51%|█████     | 255/500 [29:15<27:48,  6.81s/it, loss=1.49e+04, v_num=1]
```

Example 828 (unknown):
```unknown
Epoch 256/500:  51%|█████     | 256/500 [29:22<27:44,  6.82s/it, loss=1.49e+04, v_num=1]
```

Example 829 (unknown):
```unknown
Epoch 256/500:  51%|█████     | 256/500 [29:22<27:44,  6.82s/it, loss=1.5e+04, v_num=1]
```

Example 830 (unknown):
```unknown
Epoch 257/500:  51%|█████     | 256/500 [29:22<27:44,  6.82s/it, loss=1.5e+04, v_num=1]
```

Example 831 (unknown):
```unknown
Epoch 257/500:  51%|█████▏    | 257/500 [29:29<27:50,  6.88s/it, loss=1.5e+04, v_num=1]
```

Example 832 (unknown):
```unknown
Epoch 257/500:  51%|█████▏    | 257/500 [29:29<27:50,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 833 (unknown):
```unknown
Epoch 258/500:  51%|█████▏    | 257/500 [29:29<27:50,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 834 (unknown):
```unknown
Epoch 258/500:  52%|█████▏    | 258/500 [29:36<27:50,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 835 (unknown):
```unknown
Epoch 258/500:  52%|█████▏    | 258/500 [29:36<27:50,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 836 (unknown):
```unknown
Epoch 259/500:  52%|█████▏    | 258/500 [29:36<27:50,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 837 (unknown):
```unknown
Epoch 259/500:  52%|█████▏    | 259/500 [29:43<27:50,  6.93s/it, loss=1.51e+04, v_num=1]
```

Example 838 (unknown):
```unknown
Epoch 259/500:  52%|█████▏    | 259/500 [29:43<27:50,  6.93s/it, loss=1.53e+04, v_num=1]
```

Example 839 (unknown):
```unknown
Epoch 260/500:  52%|█████▏    | 259/500 [29:43<27:50,  6.93s/it, loss=1.53e+04, v_num=1]
```

Example 840 (unknown):
```unknown
Epoch 260/500:  52%|█████▏    | 260/500 [29:49<27:44,  6.93s/it, loss=1.53e+04, v_num=1]
```

Example 841 (unknown):
```unknown
Epoch 260/500:  52%|█████▏    | 260/500 [29:50<27:44,  6.93s/it, loss=1.52e+04, v_num=1]
```

Example 842 (unknown):
```unknown
Epoch 261/500:  52%|█████▏    | 260/500 [29:50<27:44,  6.93s/it, loss=1.52e+04, v_num=1]
```

Example 843 (unknown):
```unknown
Epoch 261/500:  52%|█████▏    | 261/500 [29:56<27:19,  6.86s/it, loss=1.52e+04, v_num=1]
```

Example 844 (unknown):
```unknown
Epoch 261/500:  52%|█████▏    | 261/500 [29:56<27:19,  6.86s/it, loss=1.51e+04, v_num=1]
```

Example 845 (unknown):
```unknown
Epoch 262/500:  52%|█████▏    | 261/500 [29:56<27:19,  6.86s/it, loss=1.51e+04, v_num=1]
```

Example 846 (unknown):
```unknown
Epoch 262/500:  52%|█████▏    | 262/500 [30:03<26:42,  6.74s/it, loss=1.51e+04, v_num=1]
```

Example 847 (unknown):
```unknown
Epoch 262/500:  52%|█████▏    | 262/500 [30:03<26:42,  6.74s/it, loss=1.52e+04, v_num=1]
```

Example 848 (unknown):
```unknown
Epoch 263/500:  52%|█████▏    | 262/500 [30:03<26:42,  6.74s/it, loss=1.52e+04, v_num=1]
```

Example 849 (unknown):
```unknown
Epoch 263/500:  53%|█████▎    | 263/500 [30:09<26:45,  6.77s/it, loss=1.52e+04, v_num=1]
```

Example 850 (unknown):
```unknown
Epoch 263/500:  53%|█████▎    | 263/500 [30:09<26:45,  6.77s/it, loss=1.5e+04, v_num=1]
```

Example 851 (unknown):
```unknown
Epoch 264/500:  53%|█████▎    | 263/500 [30:09<26:45,  6.77s/it, loss=1.5e+04, v_num=1]
```

Example 852 (unknown):
```unknown
Epoch 264/500:  53%|█████▎    | 264/500 [30:16<26:36,  6.77s/it, loss=1.5e+04, v_num=1]
```

Example 853 (unknown):
```unknown
Epoch 264/500:  53%|█████▎    | 264/500 [30:16<26:36,  6.77s/it, loss=1.51e+04, v_num=1]
```

Example 854 (unknown):
```unknown
Epoch 265/500:  53%|█████▎    | 264/500 [30:16<26:36,  6.77s/it, loss=1.51e+04, v_num=1]
```

Example 855 (unknown):
```unknown
Epoch 265/500:  53%|█████▎    | 265/500 [30:23<26:44,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 856 (unknown):
```unknown
Epoch 265/500:  53%|█████▎    | 265/500 [30:23<26:44,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 857 (unknown):
```unknown
Epoch 266/500:  53%|█████▎    | 265/500 [30:23<26:44,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 858 (unknown):
```unknown
Epoch 266/500:  53%|█████▎    | 266/500 [30:30<26:26,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 859 (unknown):
```unknown
Epoch 266/500:  53%|█████▎    | 266/500 [30:30<26:26,  6.78s/it, loss=1.5e+04, v_num=1]
```

Example 860 (unknown):
```unknown
Epoch 267/500:  53%|█████▎    | 266/500 [30:30<26:26,  6.78s/it, loss=1.5e+04, v_num=1]
```

Example 861 (unknown):
```unknown
Epoch 267/500:  53%|█████▎    | 267/500 [30:37<26:32,  6.83s/it, loss=1.5e+04, v_num=1]
```

Example 862 (unknown):
```unknown
Epoch 267/500:  53%|█████▎    | 267/500 [30:37<26:32,  6.83s/it, loss=1.52e+04, v_num=1]
```

Example 863 (unknown):
```unknown
Epoch 268/500:  53%|█████▎    | 267/500 [30:37<26:32,  6.83s/it, loss=1.52e+04, v_num=1]
```

Example 864 (unknown):
```unknown
Epoch 268/500:  54%|█████▎    | 268/500 [30:44<26:35,  6.88s/it, loss=1.52e+04, v_num=1]
```

Example 865 (unknown):
```unknown
Epoch 268/500:  54%|█████▎    | 268/500 [30:44<26:35,  6.88s/it, loss=1.48e+04, v_num=1]
```

Example 866 (unknown):
```unknown
Epoch 269/500:  54%|█████▎    | 268/500 [30:44<26:35,  6.88s/it, loss=1.48e+04, v_num=1]
```

Example 867 (unknown):
```unknown
Epoch 269/500:  54%|█████▍    | 269/500 [30:51<26:30,  6.88s/it, loss=1.48e+04, v_num=1]
```

Example 868 (unknown):
```unknown
Epoch 269/500:  54%|█████▍    | 269/500 [30:51<26:30,  6.88s/it, loss=1.5e+04, v_num=1]
```

Example 869 (unknown):
```unknown
Epoch 270/500:  54%|█████▍    | 269/500 [30:51<26:30,  6.88s/it, loss=1.5e+04, v_num=1]
```

Example 870 (unknown):
```unknown
Epoch 270/500:  54%|█████▍    | 270/500 [30:58<26:31,  6.92s/it, loss=1.5e+04, v_num=1]
```

Example 871 (unknown):
```unknown
Epoch 270/500:  54%|█████▍    | 270/500 [30:58<26:31,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 872 (unknown):
```unknown
Epoch 271/500:  54%|█████▍    | 270/500 [30:58<26:31,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 873 (unknown):
```unknown
Epoch 271/500:  54%|█████▍    | 271/500 [31:05<26:27,  6.93s/it, loss=1.51e+04, v_num=1]
```

Example 874 (unknown):
```unknown
Epoch 271/500:  54%|█████▍    | 271/500 [31:05<26:27,  6.93s/it, loss=1.47e+04, v_num=1]
```

Example 875 (unknown):
```unknown
Epoch 272/500:  54%|█████▍    | 271/500 [31:05<26:27,  6.93s/it, loss=1.47e+04, v_num=1]
```

Example 876 (unknown):
```unknown
Epoch 272/500:  54%|█████▍    | 272/500 [31:12<26:13,  6.90s/it, loss=1.47e+04, v_num=1]
```

Example 877 (unknown):
```unknown
Epoch 272/500:  54%|█████▍    | 272/500 [31:12<26:13,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 878 (unknown):
```unknown
Epoch 273/500:  54%|█████▍    | 272/500 [31:12<26:13,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 879 (unknown):
```unknown
Epoch 273/500:  55%|█████▍    | 273/500 [31:18<26:11,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 880 (unknown):
```unknown
Epoch 273/500:  55%|█████▍    | 273/500 [31:18<26:11,  6.92s/it, loss=1.5e+04, v_num=1]
```

Example 881 (unknown):
```unknown
Epoch 274/500:  55%|█████▍    | 273/500 [31:18<26:11,  6.92s/it, loss=1.5e+04, v_num=1]
```

Example 882 (unknown):
```unknown
Epoch 274/500:  55%|█████▍    | 274/500 [31:25<25:41,  6.82s/it, loss=1.5e+04, v_num=1]
```

Example 883 (unknown):
```unknown
Epoch 274/500:  55%|█████▍    | 274/500 [31:25<25:41,  6.82s/it, loss=1.5e+04, v_num=1]
```

Example 884 (unknown):
```unknown
Epoch 275/500:  55%|█████▍    | 274/500 [31:25<25:41,  6.82s/it, loss=1.5e+04, v_num=1]
```

Example 885 (unknown):
```unknown
Epoch 275/500:  55%|█████▌    | 275/500 [31:32<25:16,  6.74s/it, loss=1.5e+04, v_num=1]
```

Example 886 (unknown):
```unknown
Epoch 275/500:  55%|█████▌    | 275/500 [31:32<25:16,  6.74s/it, loss=1.48e+04, v_num=1]
```

Example 887 (unknown):
```unknown
Epoch 276/500:  55%|█████▌    | 275/500 [31:32<25:16,  6.74s/it, loss=1.48e+04, v_num=1]
```

Example 888 (unknown):
```unknown
Epoch 276/500:  55%|█████▌    | 276/500 [31:38<25:18,  6.78s/it, loss=1.48e+04, v_num=1]
```

Example 889 (unknown):
```unknown
Epoch 276/500:  55%|█████▌    | 276/500 [31:38<25:18,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 890 (unknown):
```unknown
Epoch 277/500:  55%|█████▌    | 276/500 [31:38<25:18,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 891 (unknown):
```unknown
Epoch 277/500:  55%|█████▌    | 277/500 [31:45<25:12,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 892 (unknown):
```unknown
Epoch 277/500:  55%|█████▌    | 277/500 [31:45<25:12,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 893 (unknown):
```unknown
Epoch 278/500:  55%|█████▌    | 277/500 [31:45<25:12,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 894 (unknown):
```unknown
Epoch 278/500:  56%|█████▌    | 278/500 [31:52<25:04,  6.78s/it, loss=1.51e+04, v_num=1]
```

Example 895 (unknown):
```unknown
Epoch 278/500:  56%|█████▌    | 278/500 [31:52<25:04,  6.78s/it, loss=1.5e+04, v_num=1]
```

Example 896 (unknown):
```unknown
Epoch 279/500:  56%|█████▌    | 278/500 [31:52<25:04,  6.78s/it, loss=1.5e+04, v_num=1]
```

Example 897 (unknown):
```unknown
Epoch 279/500:  56%|█████▌    | 279/500 [31:59<25:06,  6.82s/it, loss=1.5e+04, v_num=1]
```

Example 898 (unknown):
```unknown
Epoch 279/500:  56%|█████▌    | 279/500 [31:59<25:06,  6.82s/it, loss=1.5e+04, v_num=1]
```

Example 899 (unknown):
```unknown
Epoch 280/500:  56%|█████▌    | 279/500 [31:59<25:06,  6.82s/it, loss=1.5e+04, v_num=1]
```

Example 900 (unknown):
```unknown
Epoch 280/500:  56%|█████▌    | 280/500 [32:06<25:12,  6.88s/it, loss=1.5e+04, v_num=1]
```

Example 901 (unknown):
```unknown
Epoch 280/500:  56%|█████▌    | 280/500 [32:06<25:12,  6.88s/it, loss=1.48e+04, v_num=1]
```

Example 902 (unknown):
```unknown
Epoch 281/500:  56%|█████▌    | 280/500 [32:06<25:12,  6.88s/it, loss=1.48e+04, v_num=1]
```

Example 903 (unknown):
```unknown
Epoch 281/500:  56%|█████▌    | 281/500 [32:13<24:58,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 904 (unknown):
```unknown
Epoch 281/500:  56%|█████▌    | 281/500 [32:13<24:58,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 905 (unknown):
```unknown
Epoch 282/500:  56%|█████▌    | 281/500 [32:13<24:58,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 906 (unknown):
```unknown
Epoch 282/500:  56%|█████▋    | 282/500 [32:20<24:55,  6.86s/it, loss=1.52e+04, v_num=1]
```

Example 907 (unknown):
```unknown
Epoch 282/500:  56%|█████▋    | 282/500 [32:20<24:55,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 908 (unknown):
```unknown
Epoch 283/500:  56%|█████▋    | 282/500 [32:20<24:55,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 909 (unknown):
```unknown
Epoch 283/500:  57%|█████▋    | 283/500 [32:27<25:03,  6.93s/it, loss=1.5e+04, v_num=1]
```

Example 910 (unknown):
```unknown
Epoch 283/500:  57%|█████▋    | 283/500 [32:27<25:03,  6.93s/it, loss=1.51e+04, v_num=1]
```

Example 911 (unknown):
```unknown
Epoch 284/500:  57%|█████▋    | 283/500 [32:27<25:03,  6.93s/it, loss=1.51e+04, v_num=1]
```

Example 912 (unknown):
```unknown
Epoch 284/500:  57%|█████▋    | 284/500 [32:33<24:35,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 913 (unknown):
```unknown
Epoch 284/500:  57%|█████▋    | 284/500 [32:33<24:35,  6.83s/it, loss=1.53e+04, v_num=1]
```

Example 914 (unknown):
```unknown
Epoch 285/500:  57%|█████▋    | 284/500 [32:33<24:35,  6.83s/it, loss=1.53e+04, v_num=1]
```

Example 915 (unknown):
```unknown
Epoch 285/500:  57%|█████▋    | 285/500 [32:40<24:35,  6.86s/it, loss=1.53e+04, v_num=1]
```

Example 916 (unknown):
```unknown
Epoch 285/500:  57%|█████▋    | 285/500 [32:40<24:35,  6.86s/it, loss=1.49e+04, v_num=1]
```

Example 917 (unknown):
```unknown
Epoch 286/500:  57%|█████▋    | 285/500 [32:40<24:35,  6.86s/it, loss=1.49e+04, v_num=1]
```

Example 918 (unknown):
```unknown
Epoch 286/500:  57%|█████▋    | 286/500 [32:47<24:35,  6.89s/it, loss=1.49e+04, v_num=1]
```

Example 919 (unknown):
```unknown
Epoch 286/500:  57%|█████▋    | 286/500 [32:47<24:35,  6.89s/it, loss=1.47e+04, v_num=1]
```

Example 920 (unknown):
```unknown
Epoch 287/500:  57%|█████▋    | 286/500 [32:47<24:35,  6.89s/it, loss=1.47e+04, v_num=1]
```

Example 921 (unknown):
```unknown
Epoch 287/500:  57%|█████▋    | 287/500 [32:54<24:07,  6.80s/it, loss=1.47e+04, v_num=1]
```

Example 922 (unknown):
```unknown
Epoch 287/500:  57%|█████▋    | 287/500 [32:54<24:07,  6.80s/it, loss=1.5e+04, v_num=1]
```

Example 923 (unknown):
```unknown
Epoch 288/500:  57%|█████▋    | 287/500 [32:54<24:07,  6.80s/it, loss=1.5e+04, v_num=1]
```

Example 924 (unknown):
```unknown
Epoch 288/500:  58%|█████▊    | 288/500 [33:01<24:13,  6.85s/it, loss=1.5e+04, v_num=1]
```

Example 925 (unknown):
```unknown
Epoch 288/500:  58%|█████▊    | 288/500 [33:01<24:13,  6.85s/it, loss=1.5e+04, v_num=1]
```

Example 926 (unknown):
```unknown
Epoch 289/500:  58%|█████▊    | 288/500 [33:01<24:13,  6.85s/it, loss=1.5e+04, v_num=1]
```

Example 927 (unknown):
```unknown
Epoch 289/500:  58%|█████▊    | 289/500 [33:08<24:03,  6.84s/it, loss=1.5e+04, v_num=1]
```

Example 928 (unknown):
```unknown
Epoch 289/500:  58%|█████▊    | 289/500 [33:08<24:03,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 929 (unknown):
```unknown
Epoch 290/500:  58%|█████▊    | 289/500 [33:08<24:03,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 930 (unknown):
```unknown
Epoch 290/500:  58%|█████▊    | 290/500 [33:15<24:01,  6.87s/it, loss=1.49e+04, v_num=1]
```

Example 931 (unknown):
```unknown
Epoch 290/500:  58%|█████▊    | 290/500 [33:15<24:01,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 932 (unknown):
```unknown
Epoch 291/500:  58%|█████▊    | 290/500 [33:15<24:01,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 933 (unknown):
```unknown
Epoch 291/500:  58%|█████▊    | 291/500 [33:22<24:11,  6.94s/it, loss=1.48e+04, v_num=1]
```

Example 934 (unknown):
```unknown
Epoch 291/500:  58%|█████▊    | 291/500 [33:22<24:11,  6.94s/it, loss=1.5e+04, v_num=1]
```

Example 935 (unknown):
```unknown
Epoch 292/500:  58%|█████▊    | 291/500 [33:22<24:11,  6.94s/it, loss=1.5e+04, v_num=1]
```

Example 936 (unknown):
```unknown
Epoch 292/500:  58%|█████▊    | 292/500 [33:28<23:55,  6.90s/it, loss=1.5e+04, v_num=1]
```

Example 937 (unknown):
```unknown
Epoch 292/500:  58%|█████▊    | 292/500 [33:28<23:55,  6.90s/it, loss=1.48e+04, v_num=1]
```

Example 938 (unknown):
```unknown
Epoch 293/500:  58%|█████▊    | 292/500 [33:28<23:55,  6.90s/it, loss=1.48e+04, v_num=1]
```

Example 939 (unknown):
```unknown
Epoch 293/500:  59%|█████▊    | 293/500 [33:35<23:26,  6.79s/it, loss=1.48e+04, v_num=1]
```

Example 940 (unknown):
```unknown
Epoch 293/500:  59%|█████▊    | 293/500 [33:35<23:26,  6.79s/it, loss=1.52e+04, v_num=1]
```

Example 941 (unknown):
```unknown
Epoch 294/500:  59%|█████▊    | 293/500 [33:35<23:26,  6.79s/it, loss=1.52e+04, v_num=1]
```

Example 942 (unknown):
```unknown
Epoch 294/500:  59%|█████▉    | 294/500 [33:42<23:27,  6.83s/it, loss=1.52e+04, v_num=1]
```

Example 943 (unknown):
```unknown
Epoch 294/500:  59%|█████▉    | 294/500 [33:42<23:27,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 944 (unknown):
```unknown
Epoch 295/500:  59%|█████▉    | 294/500 [33:42<23:27,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 945 (unknown):
```unknown
Epoch 295/500:  59%|█████▉    | 295/500 [33:49<23:31,  6.88s/it, loss=1.51e+04, v_num=1]
```

Example 946 (unknown):
```unknown
Epoch 295/500:  59%|█████▉    | 295/500 [33:49<23:31,  6.88s/it, loss=1.49e+04, v_num=1]
```

Example 947 (unknown):
```unknown
Epoch 296/500:  59%|█████▉    | 295/500 [33:49<23:31,  6.88s/it, loss=1.49e+04, v_num=1]
```

Example 948 (unknown):
```unknown
Epoch 296/500:  59%|█████▉    | 296/500 [33:56<23:22,  6.87s/it, loss=1.49e+04, v_num=1]
```

Example 949 (unknown):
```unknown
Epoch 296/500:  59%|█████▉    | 296/500 [33:56<23:22,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 950 (unknown):
```unknown
Epoch 297/500:  59%|█████▉    | 296/500 [33:56<23:22,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 951 (unknown):
```unknown
Epoch 297/500:  59%|█████▉    | 297/500 [34:03<23:27,  6.93s/it, loss=1.48e+04, v_num=1]
```

Example 952 (unknown):
```unknown
Epoch 297/500:  59%|█████▉    | 297/500 [34:03<23:27,  6.93s/it, loss=1.5e+04, v_num=1]
```

Example 953 (unknown):
```unknown
Epoch 298/500:  59%|█████▉    | 297/500 [34:03<23:27,  6.93s/it, loss=1.5e+04, v_num=1]
```

Example 954 (unknown):
```unknown
Epoch 298/500:  60%|█████▉    | 298/500 [34:10<23:19,  6.93s/it, loss=1.5e+04, v_num=1]
```

Example 955 (unknown):
```unknown
Epoch 298/500:  60%|█████▉    | 298/500 [34:10<23:19,  6.93s/it, loss=1.49e+04, v_num=1]
```

Example 956 (unknown):
```unknown
Epoch 299/500:  60%|█████▉    | 298/500 [34:10<23:19,  6.93s/it, loss=1.49e+04, v_num=1]
```

Example 957 (unknown):
```unknown
Epoch 299/500:  60%|█████▉    | 299/500 [34:16<22:49,  6.81s/it, loss=1.49e+04, v_num=1]
```

Example 958 (unknown):
```unknown
Epoch 299/500:  60%|█████▉    | 299/500 [34:16<22:49,  6.81s/it, loss=1.47e+04, v_num=1]
```

Example 959 (unknown):
```unknown
Epoch 300/500:  60%|█████▉    | 299/500 [34:16<22:49,  6.81s/it, loss=1.47e+04, v_num=1]
```

Example 960 (unknown):
```unknown
Epoch 300/500:  60%|██████    | 300/500 [34:23<22:42,  6.81s/it, loss=1.47e+04, v_num=1]
```

Example 961 (unknown):
```unknown
Epoch 300/500:  60%|██████    | 300/500 [34:23<22:42,  6.81s/it, loss=1.49e+04, v_num=1]
```

Example 962 (unknown):
```unknown
Epoch 301/500:  60%|██████    | 300/500 [34:23<22:42,  6.81s/it, loss=1.49e+04, v_num=1]
```

Example 963 (unknown):
```unknown
Epoch 301/500:  60%|██████    | 301/500 [34:30<22:19,  6.73s/it, loss=1.49e+04, v_num=1]
```

Example 964 (unknown):
```unknown
Epoch 301/500:  60%|██████    | 301/500 [34:30<22:19,  6.73s/it, loss=1.5e+04, v_num=1]
```

Example 965 (unknown):
```unknown
Epoch 302/500:  60%|██████    | 301/500 [34:30<22:19,  6.73s/it, loss=1.5e+04, v_num=1]
```

Example 966 (unknown):
```unknown
Epoch 302/500:  60%|██████    | 302/500 [34:37<22:25,  6.80s/it, loss=1.5e+04, v_num=1]
```

Example 967 (unknown):
```unknown
Epoch 302/500:  60%|██████    | 302/500 [34:37<22:25,  6.80s/it, loss=1.48e+04, v_num=1]
```

Example 968 (unknown):
```unknown
Epoch 303/500:  60%|██████    | 302/500 [34:37<22:25,  6.80s/it, loss=1.48e+04, v_num=1]
```

Example 969 (unknown):
```unknown
Epoch 303/500:  61%|██████    | 303/500 [34:44<22:26,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 970 (unknown):
```unknown
Epoch 303/500:  61%|██████    | 303/500 [34:44<22:26,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 971 (unknown):
```unknown
Epoch 304/500:  61%|██████    | 303/500 [34:44<22:26,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 972 (unknown):
```unknown
Epoch 304/500:  61%|██████    | 304/500 [34:50<22:21,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 973 (unknown):
```unknown
Epoch 304/500:  61%|██████    | 304/500 [34:50<22:21,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 974 (unknown):
```unknown
Epoch 305/500:  61%|██████    | 304/500 [34:50<22:21,  6.84s/it, loss=1.52e+04, v_num=1]
```

Example 975 (unknown):
```unknown
Epoch 305/500:  61%|██████    | 305/500 [34:57<22:19,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 976 (unknown):
```unknown
Epoch 305/500:  61%|██████    | 305/500 [34:57<22:19,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 977 (unknown):
```unknown
Epoch 306/500:  61%|██████    | 305/500 [34:57<22:19,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 978 (unknown):
```unknown
Epoch 306/500:  61%|██████    | 306/500 [35:04<22:04,  6.83s/it, loss=1.51e+04, v_num=1]
```

Example 979 (unknown):
```unknown
Epoch 306/500:  61%|██████    | 306/500 [35:04<22:04,  6.83s/it, loss=1.48e+04, v_num=1]
```

Example 980 (unknown):
```unknown
Epoch 307/500:  61%|██████    | 306/500 [35:04<22:04,  6.83s/it, loss=1.48e+04, v_num=1]
```

Example 981 (unknown):
```unknown
Epoch 307/500:  61%|██████▏   | 307/500 [35:11<22:04,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 982 (unknown):
```unknown
Epoch 307/500:  61%|██████▏   | 307/500 [35:11<22:04,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 983 (unknown):
```unknown
Epoch 308/500:  61%|██████▏   | 307/500 [35:11<22:04,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 984 (unknown):
```unknown
Epoch 308/500:  62%|██████▏   | 308/500 [35:18<21:45,  6.80s/it, loss=1.48e+04, v_num=1]
```

Example 985 (unknown):
```unknown
Epoch 308/500:  62%|██████▏   | 308/500 [35:18<21:45,  6.80s/it, loss=1.52e+04, v_num=1]
```

Example 986 (unknown):
```unknown
Epoch 309/500:  62%|██████▏   | 308/500 [35:18<21:45,  6.80s/it, loss=1.52e+04, v_num=1]
```

Example 987 (unknown):
```unknown
Epoch 309/500:  62%|██████▏   | 309/500 [35:25<21:49,  6.85s/it, loss=1.52e+04, v_num=1]
```

Example 988 (unknown):
```unknown
Epoch 309/500:  62%|██████▏   | 309/500 [35:25<21:49,  6.85s/it, loss=1.51e+04, v_num=1]
```

Example 989 (unknown):
```unknown
Epoch 310/500:  62%|██████▏   | 309/500 [35:25<21:49,  6.85s/it, loss=1.51e+04, v_num=1]
```

Example 990 (unknown):
```unknown
Epoch 310/500:  62%|██████▏   | 310/500 [35:32<21:44,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 991 (unknown):
```unknown
Epoch 310/500:  62%|██████▏   | 310/500 [35:32<21:44,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 992 (unknown):
```unknown
Epoch 311/500:  62%|██████▏   | 310/500 [35:32<21:44,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 993 (unknown):
```unknown
Epoch 311/500:  62%|██████▏   | 311/500 [35:38<21:35,  6.85s/it, loss=1.48e+04, v_num=1]
```

Example 994 (unknown):
```unknown
Epoch 311/500:  62%|██████▏   | 311/500 [35:38<21:35,  6.85s/it, loss=1.51e+04, v_num=1]
```

Example 995 (unknown):
```unknown
Epoch 312/500:  62%|██████▏   | 311/500 [35:38<21:35,  6.85s/it, loss=1.51e+04, v_num=1]
```

Example 996 (unknown):
```unknown
Epoch 312/500:  62%|██████▏   | 312/500 [35:45<21:32,  6.87s/it, loss=1.51e+04, v_num=1]
```

Example 997 (unknown):
```unknown
Epoch 312/500:  62%|██████▏   | 312/500 [35:45<21:32,  6.87s/it, loss=1.53e+04, v_num=1]
```

Example 998 (unknown):
```unknown
Epoch 313/500:  62%|██████▏   | 312/500 [35:45<21:32,  6.87s/it, loss=1.53e+04, v_num=1]
```

Example 999 (unknown):
```unknown
Epoch 313/500:  63%|██████▎   | 313/500 [35:52<21:32,  6.91s/it, loss=1.53e+04, v_num=1]
```

Example 1000 (unknown):
```unknown
Epoch 313/500:  63%|██████▎   | 313/500 [35:52<21:32,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 1001 (unknown):
```unknown
Epoch 314/500:  63%|██████▎   | 313/500 [35:52<21:32,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 1002 (unknown):
```unknown
Epoch 314/500:  63%|██████▎   | 314/500 [35:59<21:11,  6.83s/it, loss=1.5e+04, v_num=1]
```

Example 1003 (unknown):
```unknown
Epoch 314/500:  63%|██████▎   | 314/500 [35:59<21:11,  6.83s/it, loss=1.5e+04, v_num=1]
```

Example 1004 (unknown):
```unknown
Epoch 315/500:  63%|██████▎   | 314/500 [35:59<21:11,  6.83s/it, loss=1.5e+04, v_num=1]
```

Example 1005 (unknown):
```unknown
Epoch 315/500:  63%|██████▎   | 315/500 [36:06<21:09,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 1006 (unknown):
```unknown
Epoch 315/500:  63%|██████▎   | 315/500 [36:06<21:09,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 1007 (unknown):
```unknown
Epoch 316/500:  63%|██████▎   | 315/500 [36:06<21:09,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 1008 (unknown):
```unknown
Epoch 316/500:  63%|██████▎   | 316/500 [36:13<21:01,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 1009 (unknown):
```unknown
Epoch 316/500:  63%|██████▎   | 316/500 [36:13<21:01,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 1010 (unknown):
```unknown
Epoch 317/500:  63%|██████▎   | 316/500 [36:13<21:01,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 1011 (unknown):
```unknown
Epoch 317/500:  63%|██████▎   | 317/500 [36:20<21:06,  6.92s/it, loss=1.5e+04, v_num=1]
```

Example 1012 (unknown):
```unknown
Epoch 317/500:  63%|██████▎   | 317/500 [36:20<21:06,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 1013 (unknown):
```unknown
Epoch 318/500:  63%|██████▎   | 317/500 [36:20<21:06,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 1014 (unknown):
```unknown
Epoch 318/500:  64%|██████▎   | 318/500 [36:26<20:49,  6.86s/it, loss=1.51e+04, v_num=1]
```

Example 1015 (unknown):
```unknown
Epoch 318/500:  64%|██████▎   | 318/500 [36:26<20:49,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1016 (unknown):
```unknown
Epoch 319/500:  64%|██████▎   | 318/500 [36:26<20:49,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1017 (unknown):
```unknown
Epoch 319/500:  64%|██████▍   | 319/500 [36:33<20:27,  6.78s/it, loss=1.47e+04, v_num=1]
```

Example 1018 (unknown):
```unknown
Epoch 319/500:  64%|██████▍   | 319/500 [36:33<20:27,  6.78s/it, loss=1.52e+04, v_num=1]
```

Example 1019 (unknown):
```unknown
Epoch 320/500:  64%|██████▍   | 319/500 [36:33<20:27,  6.78s/it, loss=1.52e+04, v_num=1]
```

Example 1020 (unknown):
```unknown
Epoch 320/500:  64%|██████▍   | 320/500 [36:40<20:33,  6.85s/it, loss=1.52e+04, v_num=1]
```

Example 1021 (unknown):
```unknown
Epoch 320/500:  64%|██████▍   | 320/500 [36:40<20:33,  6.85s/it, loss=1.49e+04, v_num=1]
```

Example 1022 (unknown):
```unknown
Epoch 321/500:  64%|██████▍   | 320/500 [36:40<20:33,  6.85s/it, loss=1.49e+04, v_num=1]
```

Example 1023 (unknown):
```unknown
Epoch 321/500:  64%|██████▍   | 321/500 [36:47<20:34,  6.90s/it, loss=1.49e+04, v_num=1]
```

Example 1024 (unknown):
```unknown
Epoch 321/500:  64%|██████▍   | 321/500 [36:47<20:34,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 1025 (unknown):
```unknown
Epoch 322/500:  64%|██████▍   | 321/500 [36:47<20:34,  6.90s/it, loss=1.51e+04, v_num=1]
```

Example 1026 (unknown):
```unknown
Epoch 322/500:  64%|██████▍   | 322/500 [36:54<20:21,  6.86s/it, loss=1.51e+04, v_num=1]
```

Example 1027 (unknown):
```unknown
Epoch 322/500:  64%|██████▍   | 322/500 [36:54<20:21,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 1028 (unknown):
```unknown
Epoch 323/500:  64%|██████▍   | 322/500 [36:54<20:21,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 1029 (unknown):
```unknown
Epoch 323/500:  65%|██████▍   | 323/500 [37:01<20:21,  6.90s/it, loss=1.5e+04, v_num=1]
```

Example 1030 (unknown):
```unknown
Epoch 323/500:  65%|██████▍   | 323/500 [37:01<20:21,  6.90s/it, loss=1.5e+04, v_num=1]
```

Example 1031 (unknown):
```unknown
Epoch 324/500:  65%|██████▍   | 323/500 [37:01<20:21,  6.90s/it, loss=1.5e+04, v_num=1]
```

Example 1032 (unknown):
```unknown
Epoch 324/500:  65%|██████▍   | 324/500 [37:07<19:38,  6.70s/it, loss=1.5e+04, v_num=1]
```

Example 1033 (unknown):
```unknown
Epoch 324/500:  65%|██████▍   | 324/500 [37:07<19:38,  6.70s/it, loss=1.48e+04, v_num=1]
```

Example 1034 (unknown):
```unknown
Epoch 325/500:  65%|██████▍   | 324/500 [37:07<19:38,  6.70s/it, loss=1.48e+04, v_num=1]
```

Example 1035 (unknown):
```unknown
Epoch 325/500:  65%|██████▌   | 325/500 [37:14<19:18,  6.62s/it, loss=1.48e+04, v_num=1]
```

Example 1036 (unknown):
```unknown
Epoch 325/500:  65%|██████▌   | 325/500 [37:14<19:18,  6.62s/it, loss=1.5e+04, v_num=1]
```

Example 1037 (unknown):
```unknown
Epoch 326/500:  65%|██████▌   | 325/500 [37:14<19:18,  6.62s/it, loss=1.5e+04, v_num=1]
```

Example 1038 (unknown):
```unknown
Epoch 326/500:  65%|██████▌   | 326/500 [37:21<19:32,  6.74s/it, loss=1.5e+04, v_num=1]
```

Example 1039 (unknown):
```unknown
Epoch 326/500:  65%|██████▌   | 326/500 [37:21<19:32,  6.74s/it, loss=1.5e+04, v_num=1]
```

Example 1040 (unknown):
```unknown
Epoch 327/500:  65%|██████▌   | 326/500 [37:21<19:32,  6.74s/it, loss=1.5e+04, v_num=1]
```

Example 1041 (unknown):
```unknown
Epoch 327/500:  65%|██████▌   | 327/500 [37:28<19:37,  6.81s/it, loss=1.5e+04, v_num=1]
```

Example 1042 (unknown):
```unknown
Epoch 327/500:  65%|██████▌   | 327/500 [37:28<19:37,  6.81s/it, loss=1.48e+04, v_num=1]
```

Example 1043 (unknown):
```unknown
Epoch 328/500:  65%|██████▌   | 327/500 [37:28<19:37,  6.81s/it, loss=1.48e+04, v_num=1]
```

Example 1044 (unknown):
```unknown
Epoch 328/500:  66%|██████▌   | 328/500 [37:35<19:40,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 1045 (unknown):
```unknown
Epoch 328/500:  66%|██████▌   | 328/500 [37:35<19:40,  6.86s/it, loss=1.49e+04, v_num=1]
```

Example 1046 (unknown):
```unknown
Epoch 329/500:  66%|██████▌   | 328/500 [37:35<19:40,  6.86s/it, loss=1.49e+04, v_num=1]
```

Example 1047 (unknown):
```unknown
Epoch 329/500:  66%|██████▌   | 329/500 [37:41<19:34,  6.87s/it, loss=1.49e+04, v_num=1]
```

Example 1048 (unknown):
```unknown
Epoch 329/500:  66%|██████▌   | 329/500 [37:41<19:34,  6.87s/it, loss=1.49e+04, v_num=1]
```

Example 1049 (unknown):
```unknown
Epoch 330/500:  66%|██████▌   | 329/500 [37:41<19:34,  6.87s/it, loss=1.49e+04, v_num=1]
```

Example 1050 (unknown):
```unknown
Epoch 330/500:  66%|██████▌   | 330/500 [37:48<19:28,  6.87s/it, loss=1.49e+04, v_num=1]
```

Example 1051 (unknown):
```unknown
Epoch 330/500:  66%|██████▌   | 330/500 [37:48<19:28,  6.87s/it, loss=1.47e+04, v_num=1]
```

Example 1052 (unknown):
```unknown
Epoch 331/500:  66%|██████▌   | 330/500 [37:48<19:28,  6.87s/it, loss=1.47e+04, v_num=1]
```

Example 1053 (unknown):
```unknown
Epoch 331/500:  66%|██████▌   | 331/500 [37:55<19:16,  6.84s/it, loss=1.47e+04, v_num=1]
```

Example 1054 (unknown):
```unknown
Epoch 331/500:  66%|██████▌   | 331/500 [37:55<19:16,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 1055 (unknown):
```unknown
Epoch 332/500:  66%|██████▌   | 331/500 [37:55<19:16,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 1056 (unknown):
```unknown
Epoch 332/500:  66%|██████▋   | 332/500 [38:02<19:03,  6.81s/it, loss=1.48e+04, v_num=1]
```

Example 1057 (unknown):
```unknown
Epoch 332/500:  66%|██████▋   | 332/500 [38:02<19:03,  6.81s/it, loss=1.5e+04, v_num=1]
```

Example 1058 (unknown):
```unknown
Epoch 333/500:  66%|██████▋   | 332/500 [38:02<19:03,  6.81s/it, loss=1.5e+04, v_num=1]
```

Example 1059 (unknown):
```unknown
Epoch 333/500:  67%|██████▋   | 333/500 [38:08<18:41,  6.71s/it, loss=1.5e+04, v_num=1]
```

Example 1060 (unknown):
```unknown
Epoch 333/500:  67%|██████▋   | 333/500 [38:08<18:41,  6.71s/it, loss=1.5e+04, v_num=1]
```

Example 1061 (unknown):
```unknown
Epoch 334/500:  67%|██████▋   | 333/500 [38:08<18:41,  6.71s/it, loss=1.5e+04, v_num=1]
```

Example 1062 (unknown):
```unknown
Epoch 334/500:  67%|██████▋   | 334/500 [38:15<18:47,  6.79s/it, loss=1.5e+04, v_num=1]
```

Example 1063 (unknown):
```unknown
Epoch 334/500:  67%|██████▋   | 334/500 [38:15<18:47,  6.79s/it, loss=1.48e+04, v_num=1]
```

Example 1064 (unknown):
```unknown
Epoch 335/500:  67%|██████▋   | 334/500 [38:15<18:47,  6.79s/it, loss=1.48e+04, v_num=1]
```

Example 1065 (unknown):
```unknown
Epoch 335/500:  67%|██████▋   | 335/500 [38:22<18:49,  6.85s/it, loss=1.48e+04, v_num=1]
```

Example 1066 (unknown):
```unknown
Epoch 335/500:  67%|██████▋   | 335/500 [38:22<18:49,  6.85s/it, loss=1.5e+04, v_num=1]
```

Example 1067 (unknown):
```unknown
Epoch 336/500:  67%|██████▋   | 335/500 [38:22<18:49,  6.85s/it, loss=1.5e+04, v_num=1]
```

Example 1068 (unknown):
```unknown
Epoch 336/500:  67%|██████▋   | 336/500 [38:29<18:40,  6.83s/it, loss=1.5e+04, v_num=1]
```

Example 1069 (unknown):
```unknown
Epoch 336/500:  67%|██████▋   | 336/500 [38:29<18:40,  6.83s/it, loss=1.48e+04, v_num=1]
```

Example 1070 (unknown):
```unknown
Epoch 337/500:  67%|██████▋   | 336/500 [38:29<18:40,  6.83s/it, loss=1.48e+04, v_num=1]
```

Example 1071 (unknown):
```unknown
Epoch 337/500:  67%|██████▋   | 337/500 [38:36<18:27,  6.80s/it, loss=1.48e+04, v_num=1]
```

Example 1072 (unknown):
```unknown
Epoch 337/500:  67%|██████▋   | 337/500 [38:36<18:27,  6.80s/it, loss=1.5e+04, v_num=1]
```

Example 1073 (unknown):
```unknown
Epoch 338/500:  67%|██████▋   | 337/500 [38:36<18:27,  6.80s/it, loss=1.5e+04, v_num=1]
```

Example 1074 (unknown):
```unknown
Epoch 338/500:  68%|██████▊   | 338/500 [38:43<18:28,  6.84s/it, loss=1.5e+04, v_num=1]
```

Example 1075 (unknown):
```unknown
Epoch 338/500:  68%|██████▊   | 338/500 [38:43<18:28,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 1076 (unknown):
```unknown
Epoch 339/500:  68%|██████▊   | 338/500 [38:43<18:28,  6.84s/it, loss=1.51e+04, v_num=1]
```

Example 1077 (unknown):
```unknown
Epoch 339/500:  68%|██████▊   | 339/500 [38:50<18:22,  6.85s/it, loss=1.51e+04, v_num=1]
```

Example 1078 (unknown):
```unknown
Epoch 339/500:  68%|██████▊   | 339/500 [38:50<18:22,  6.85s/it, loss=1.49e+04, v_num=1]
```

Example 1079 (unknown):
```unknown
Epoch 340/500:  68%|██████▊   | 339/500 [38:50<18:22,  6.85s/it, loss=1.49e+04, v_num=1]
```

Example 1080 (unknown):
```unknown
Epoch 340/500:  68%|██████▊   | 340/500 [38:56<18:15,  6.85s/it, loss=1.49e+04, v_num=1]
```

Example 1081 (unknown):
```unknown
Epoch 340/500:  68%|██████▊   | 340/500 [38:56<18:15,  6.85s/it, loss=1.48e+04, v_num=1]
```

Example 1082 (unknown):
```unknown
Epoch 341/500:  68%|██████▊   | 340/500 [38:56<18:15,  6.85s/it, loss=1.48e+04, v_num=1]
```

Example 1083 (unknown):
```unknown
Epoch 341/500:  68%|██████▊   | 341/500 [39:03<18:09,  6.85s/it, loss=1.48e+04, v_num=1]
```

Example 1084 (unknown):
```unknown
Epoch 341/500:  68%|██████▊   | 341/500 [39:03<18:09,  6.85s/it, loss=1.49e+04, v_num=1]
```

Example 1085 (unknown):
```unknown
Epoch 342/500:  68%|██████▊   | 341/500 [39:03<18:09,  6.85s/it, loss=1.49e+04, v_num=1]
```

Example 1086 (unknown):
```unknown
Epoch 342/500:  68%|██████▊   | 342/500 [39:10<18:01,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 1087 (unknown):
```unknown
Epoch 342/500:  68%|██████▊   | 342/500 [39:10<18:01,  6.84s/it, loss=1.47e+04, v_num=1]
```

Example 1088 (unknown):
```unknown
Epoch 343/500:  68%|██████▊   | 342/500 [39:10<18:01,  6.84s/it, loss=1.47e+04, v_num=1]
```

Example 1089 (unknown):
```unknown
Epoch 343/500:  69%|██████▊   | 343/500 [39:17<17:48,  6.81s/it, loss=1.47e+04, v_num=1]
```

Example 1090 (unknown):
```unknown
Epoch 343/500:  69%|██████▊   | 343/500 [39:17<17:48,  6.81s/it, loss=1.5e+04, v_num=1]
```

Example 1091 (unknown):
```unknown
Epoch 344/500:  69%|██████▊   | 343/500 [39:17<17:48,  6.81s/it, loss=1.5e+04, v_num=1]
```

Example 1092 (unknown):
```unknown
Epoch 344/500:  69%|██████▉   | 344/500 [39:24<17:48,  6.85s/it, loss=1.5e+04, v_num=1]
```

Example 1093 (unknown):
```unknown
Epoch 344/500:  69%|██████▉   | 344/500 [39:24<17:48,  6.85s/it, loss=1.49e+04, v_num=1]
```

Example 1094 (unknown):
```unknown
Epoch 345/500:  69%|██████▉   | 344/500 [39:24<17:48,  6.85s/it, loss=1.49e+04, v_num=1]
```

Example 1095 (unknown):
```unknown
Epoch 345/500:  69%|██████▉   | 345/500 [39:30<17:30,  6.78s/it, loss=1.49e+04, v_num=1]
```

Example 1096 (unknown):
```unknown
Epoch 345/500:  69%|██████▉   | 345/500 [39:30<17:30,  6.78s/it, loss=1.47e+04, v_num=1]
```

Example 1097 (unknown):
```unknown
Epoch 346/500:  69%|██████▉   | 345/500 [39:30<17:30,  6.78s/it, loss=1.47e+04, v_num=1]
```

Example 1098 (unknown):
```unknown
Epoch 346/500:  69%|██████▉   | 346/500 [39:37<17:30,  6.82s/it, loss=1.47e+04, v_num=1]
```

Example 1099 (unknown):
```unknown
Epoch 346/500:  69%|██████▉   | 346/500 [39:37<17:30,  6.82s/it, loss=1.48e+04, v_num=1]
```

Example 1100 (unknown):
```unknown
Epoch 347/500:  69%|██████▉   | 346/500 [39:37<17:30,  6.82s/it, loss=1.48e+04, v_num=1]
```

Example 1101 (unknown):
```unknown
Epoch 347/500:  69%|██████▉   | 347/500 [39:44<17:21,  6.81s/it, loss=1.48e+04, v_num=1]
```

Example 1102 (unknown):
```unknown
Epoch 347/500:  69%|██████▉   | 347/500 [39:44<17:21,  6.81s/it, loss=1.49e+04, v_num=1]
```

Example 1103 (unknown):
```unknown
Epoch 348/500:  69%|██████▉   | 347/500 [39:44<17:21,  6.81s/it, loss=1.49e+04, v_num=1]
```

Example 1104 (unknown):
```unknown
Epoch 348/500:  70%|██████▉   | 348/500 [39:51<17:22,  6.86s/it, loss=1.49e+04, v_num=1]
```

Example 1105 (unknown):
```unknown
Epoch 348/500:  70%|██████▉   | 348/500 [39:51<17:22,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1106 (unknown):
```unknown
Epoch 349/500:  70%|██████▉   | 348/500 [39:51<17:22,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1107 (unknown):
```unknown
Epoch 349/500:  70%|██████▉   | 349/500 [39:58<17:20,  6.89s/it, loss=1.47e+04, v_num=1]
```

Example 1108 (unknown):
```unknown
Epoch 349/500:  70%|██████▉   | 349/500 [39:58<17:20,  6.89s/it, loss=1.5e+04, v_num=1]
```

Example 1109 (unknown):
```unknown
Epoch 350/500:  70%|██████▉   | 349/500 [39:58<17:20,  6.89s/it, loss=1.5e+04, v_num=1]
```

Example 1110 (unknown):
```unknown
Epoch 350/500:  70%|███████   | 350/500 [40:05<17:13,  6.89s/it, loss=1.5e+04, v_num=1]
```

Example 1111 (unknown):
```unknown
Epoch 350/500:  70%|███████   | 350/500 [40:05<17:13,  6.89s/it, loss=1.47e+04, v_num=1]
```

Example 1112 (unknown):
```unknown
Epoch 351/500:  70%|███████   | 350/500 [40:05<17:13,  6.89s/it, loss=1.47e+04, v_num=1]
```

Example 1113 (unknown):
```unknown
Epoch 351/500:  70%|███████   | 351/500 [40:12<17:10,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1114 (unknown):
```unknown
Epoch 351/500:  70%|███████   | 351/500 [40:12<17:10,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1115 (unknown):
```unknown
Epoch 352/500:  70%|███████   | 351/500 [40:12<17:10,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1116 (unknown):
```unknown
Epoch 352/500:  70%|███████   | 352/500 [40:19<17:09,  6.96s/it, loss=1.47e+04, v_num=1]
```

Example 1117 (unknown):
```unknown
Epoch 352/500:  70%|███████   | 352/500 [40:19<17:09,  6.96s/it, loss=1.47e+04, v_num=1]
```

Example 1118 (unknown):
```unknown
Epoch 353/500:  70%|███████   | 352/500 [40:19<17:09,  6.96s/it, loss=1.47e+04, v_num=1]
```

Example 1119 (unknown):
```unknown
Epoch 353/500:  71%|███████   | 353/500 [40:26<16:53,  6.89s/it, loss=1.47e+04, v_num=1]
```

Example 1120 (unknown):
```unknown
Epoch 353/500:  71%|███████   | 353/500 [40:26<16:53,  6.89s/it, loss=1.49e+04, v_num=1]
```

Example 1121 (unknown):
```unknown
Epoch 354/500:  71%|███████   | 353/500 [40:26<16:53,  6.89s/it, loss=1.49e+04, v_num=1]
```

Example 1122 (unknown):
```unknown
Epoch 354/500:  71%|███████   | 354/500 [40:33<16:49,  6.92s/it, loss=1.49e+04, v_num=1]
```

Example 1123 (unknown):
```unknown
Epoch 354/500:  71%|███████   | 354/500 [40:33<16:49,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1124 (unknown):
```unknown
Epoch 355/500:  71%|███████   | 354/500 [40:33<16:49,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1125 (unknown):
```unknown
Epoch 355/500:  71%|███████   | 355/500 [40:40<16:43,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1126 (unknown):
```unknown
Epoch 355/500:  71%|███████   | 355/500 [40:40<16:43,  6.92s/it, loss=1.48e+04, v_num=1]
```

Example 1127 (unknown):
```unknown
Epoch 356/500:  71%|███████   | 355/500 [40:40<16:43,  6.92s/it, loss=1.48e+04, v_num=1]
```

Example 1128 (unknown):
```unknown
Epoch 356/500:  71%|███████   | 356/500 [40:46<16:35,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1129 (unknown):
```unknown
Epoch 356/500:  71%|███████   | 356/500 [40:46<16:35,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1130 (unknown):
```unknown
Epoch 357/500:  71%|███████   | 356/500 [40:46<16:35,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1131 (unknown):
```unknown
Epoch 357/500:  71%|███████▏  | 357/500 [40:53<16:07,  6.77s/it, loss=1.48e+04, v_num=1]
```

Example 1132 (unknown):
```unknown
Epoch 357/500:  71%|███████▏  | 357/500 [40:53<16:07,  6.77s/it, loss=1.47e+04, v_num=1]
```

Example 1133 (unknown):
```unknown
Epoch 358/500:  71%|███████▏  | 357/500 [40:53<16:07,  6.77s/it, loss=1.47e+04, v_num=1]
```

Example 1134 (unknown):
```unknown
Epoch 358/500:  72%|███████▏  | 358/500 [40:59<15:51,  6.70s/it, loss=1.47e+04, v_num=1]
```

Example 1135 (unknown):
```unknown
Epoch 358/500:  72%|███████▏  | 358/500 [40:59<15:51,  6.70s/it, loss=1.49e+04, v_num=1]
```

Example 1136 (unknown):
```unknown
Epoch 359/500:  72%|███████▏  | 358/500 [40:59<15:51,  6.70s/it, loss=1.49e+04, v_num=1]
```

Example 1137 (unknown):
```unknown
Epoch 359/500:  72%|███████▏  | 359/500 [41:06<15:54,  6.77s/it, loss=1.49e+04, v_num=1]
```

Example 1138 (unknown):
```unknown
Epoch 359/500:  72%|███████▏  | 359/500 [41:06<15:54,  6.77s/it, loss=1.47e+04, v_num=1]
```

Example 1139 (unknown):
```unknown
Epoch 360/500:  72%|███████▏  | 359/500 [41:06<15:54,  6.77s/it, loss=1.47e+04, v_num=1]
```

Example 1140 (unknown):
```unknown
Epoch 360/500:  72%|███████▏  | 360/500 [41:13<15:56,  6.83s/it, loss=1.47e+04, v_num=1]
```

Example 1141 (unknown):
```unknown
Epoch 360/500:  72%|███████▏  | 360/500 [41:13<15:56,  6.83s/it, loss=1.47e+04, v_num=1]
```

Example 1142 (unknown):
```unknown
Epoch 361/500:  72%|███████▏  | 360/500 [41:13<15:56,  6.83s/it, loss=1.47e+04, v_num=1]
```

Example 1143 (unknown):
```unknown
Epoch 361/500:  72%|███████▏  | 361/500 [41:20<15:53,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1144 (unknown):
```unknown
Epoch 361/500:  72%|███████▏  | 361/500 [41:20<15:53,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 1145 (unknown):
```unknown
Epoch 362/500:  72%|███████▏  | 361/500 [41:20<15:53,  6.86s/it, loss=1.5e+04, v_num=1]
```

Example 1146 (unknown):
```unknown
Epoch 362/500:  72%|███████▏  | 362/500 [41:27<15:49,  6.88s/it, loss=1.5e+04, v_num=1]
```

Example 1147 (unknown):
```unknown
Epoch 362/500:  72%|███████▏  | 362/500 [41:27<15:49,  6.88s/it, loss=1.49e+04, v_num=1]
```

Example 1148 (unknown):
```unknown
Epoch 363/500:  72%|███████▏  | 362/500 [41:27<15:49,  6.88s/it, loss=1.49e+04, v_num=1]
```

Example 1149 (unknown):
```unknown
Epoch 363/500:  73%|███████▎  | 363/500 [41:34<15:48,  6.92s/it, loss=1.49e+04, v_num=1]
```

Example 1150 (unknown):
```unknown
Epoch 363/500:  73%|███████▎  | 363/500 [41:34<15:48,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1151 (unknown):
```unknown
Epoch 364/500:  73%|███████▎  | 363/500 [41:34<15:48,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1152 (unknown):
```unknown
Epoch 364/500:  73%|███████▎  | 364/500 [41:41<15:44,  6.95s/it, loss=1.47e+04, v_num=1]
```

Example 1153 (unknown):
```unknown
Epoch 364/500:  73%|███████▎  | 364/500 [41:41<15:44,  6.95s/it, loss=1.49e+04, v_num=1]
```

Example 1154 (unknown):
```unknown
Epoch 365/500:  73%|███████▎  | 364/500 [41:41<15:44,  6.95s/it, loss=1.49e+04, v_num=1]
```

Example 1155 (unknown):
```unknown
Epoch 365/500:  73%|███████▎  | 365/500 [41:48<15:33,  6.91s/it, loss=1.49e+04, v_num=1]
```

Example 1156 (unknown):
```unknown
Epoch 365/500:  73%|███████▎  | 365/500 [41:48<15:33,  6.91s/it, loss=1.49e+04, v_num=1]
```

Example 1157 (unknown):
```unknown
Epoch 366/500:  73%|███████▎  | 365/500 [41:48<15:33,  6.91s/it, loss=1.49e+04, v_num=1]
```

Example 1158 (unknown):
```unknown
Epoch 366/500:  73%|███████▎  | 366/500 [41:55<15:28,  6.93s/it, loss=1.49e+04, v_num=1]
```

Example 1159 (unknown):
```unknown
Epoch 366/500:  73%|███████▎  | 366/500 [41:55<15:28,  6.93s/it, loss=1.47e+04, v_num=1]
```

Example 1160 (unknown):
```unknown
Epoch 367/500:  73%|███████▎  | 366/500 [41:55<15:28,  6.93s/it, loss=1.47e+04, v_num=1]
```

Example 1161 (unknown):
```unknown
Epoch 367/500:  73%|███████▎  | 367/500 [42:02<15:28,  6.98s/it, loss=1.47e+04, v_num=1]
```

Example 1162 (unknown):
```unknown
Epoch 367/500:  73%|███████▎  | 367/500 [42:02<15:28,  6.98s/it, loss=1.46e+04, v_num=1]
```

Example 1163 (unknown):
```unknown
Epoch 368/500:  73%|███████▎  | 367/500 [42:02<15:28,  6.98s/it, loss=1.46e+04, v_num=1]
```

Example 1164 (unknown):
```unknown
Epoch 368/500:  74%|███████▎  | 368/500 [42:09<15:14,  6.93s/it, loss=1.46e+04, v_num=1]
```

Example 1165 (unknown):
```unknown
Epoch 368/500:  74%|███████▎  | 368/500 [42:09<15:14,  6.93s/it, loss=1.52e+04, v_num=1]
```

Example 1166 (unknown):
```unknown
Epoch 369/500:  74%|███████▎  | 368/500 [42:09<15:14,  6.93s/it, loss=1.52e+04, v_num=1]
```

Example 1167 (unknown):
```unknown
Epoch 369/500:  74%|███████▍  | 369/500 [42:16<14:59,  6.87s/it, loss=1.52e+04, v_num=1]
```

Example 1168 (unknown):
```unknown
Epoch 369/500:  74%|███████▍  | 369/500 [42:16<14:59,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 1169 (unknown):
```unknown
Epoch 370/500:  74%|███████▍  | 369/500 [42:16<14:59,  6.87s/it, loss=1.48e+04, v_num=1]
```

Example 1170 (unknown):
```unknown
Epoch 370/500:  74%|███████▍  | 370/500 [42:22<14:49,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 1171 (unknown):
```unknown
Epoch 370/500:  74%|███████▍  | 370/500 [42:22<14:49,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 1172 (unknown):
```unknown
Epoch 371/500:  74%|███████▍  | 370/500 [42:22<14:49,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 1173 (unknown):
```unknown
Epoch 371/500:  74%|███████▍  | 371/500 [42:29<14:31,  6.75s/it, loss=1.49e+04, v_num=1]
```

Example 1174 (unknown):
```unknown
Epoch 371/500:  74%|███████▍  | 371/500 [42:29<14:31,  6.75s/it, loss=1.48e+04, v_num=1]
```

Example 1175 (unknown):
```unknown
Epoch 372/500:  74%|███████▍  | 371/500 [42:29<14:31,  6.75s/it, loss=1.48e+04, v_num=1]
```

Example 1176 (unknown):
```unknown
Epoch 372/500:  74%|███████▍  | 372/500 [42:35<14:08,  6.63s/it, loss=1.48e+04, v_num=1]
```

Example 1177 (unknown):
```unknown
Epoch 372/500:  74%|███████▍  | 372/500 [42:35<14:08,  6.63s/it, loss=1.43e+04, v_num=1]
```

Example 1178 (unknown):
```unknown
Epoch 373/500:  74%|███████▍  | 372/500 [42:35<14:08,  6.63s/it, loss=1.43e+04, v_num=1]
```

Example 1179 (unknown):
```unknown
Epoch 373/500:  75%|███████▍  | 373/500 [42:42<14:15,  6.73s/it, loss=1.43e+04, v_num=1]
```

Example 1180 (unknown):
```unknown
Epoch 373/500:  75%|███████▍  | 373/500 [42:42<14:15,  6.73s/it, loss=1.48e+04, v_num=1]
```

Example 1181 (unknown):
```unknown
Epoch 374/500:  75%|███████▍  | 373/500 [42:42<14:15,  6.73s/it, loss=1.48e+04, v_num=1]
```

Example 1182 (unknown):
```unknown
Epoch 374/500:  75%|███████▍  | 374/500 [42:49<14:07,  6.73s/it, loss=1.48e+04, v_num=1]
```

Example 1183 (unknown):
```unknown
Epoch 374/500:  75%|███████▍  | 374/500 [42:49<14:07,  6.73s/it, loss=1.49e+04, v_num=1]
```

Example 1184 (unknown):
```unknown
Epoch 375/500:  75%|███████▍  | 374/500 [42:49<14:07,  6.73s/it, loss=1.49e+04, v_num=1]
```

Example 1185 (unknown):
```unknown
Epoch 375/500:  75%|███████▌  | 375/500 [42:56<14:01,  6.73s/it, loss=1.49e+04, v_num=1]
```

Example 1186 (unknown):
```unknown
Epoch 375/500:  75%|███████▌  | 375/500 [42:56<14:01,  6.73s/it, loss=1.5e+04, v_num=1]
```

Example 1187 (unknown):
```unknown
Epoch 376/500:  75%|███████▌  | 375/500 [42:56<14:01,  6.73s/it, loss=1.5e+04, v_num=1]
```

Example 1188 (unknown):
```unknown
Epoch 376/500:  75%|███████▌  | 376/500 [43:03<14:02,  6.79s/it, loss=1.5e+04, v_num=1]
```

Example 1189 (unknown):
```unknown
Epoch 376/500:  75%|███████▌  | 376/500 [43:03<14:02,  6.79s/it, loss=1.46e+04, v_num=1]
```

Example 1190 (unknown):
```unknown
Epoch 377/500:  75%|███████▌  | 376/500 [43:03<14:02,  6.79s/it, loss=1.46e+04, v_num=1]
```

Example 1191 (unknown):
```unknown
Epoch 377/500:  75%|███████▌  | 377/500 [43:09<13:48,  6.73s/it, loss=1.46e+04, v_num=1]
```

Example 1192 (unknown):
```unknown
Epoch 377/500:  75%|███████▌  | 377/500 [43:09<13:48,  6.73s/it, loss=1.46e+04, v_num=1]
```

Example 1193 (unknown):
```unknown
Epoch 378/500:  75%|███████▌  | 377/500 [43:09<13:48,  6.73s/it, loss=1.46e+04, v_num=1]
```

Example 1194 (unknown):
```unknown
Epoch 378/500:  76%|███████▌  | 378/500 [43:16<13:48,  6.79s/it, loss=1.46e+04, v_num=1]
```

Example 1195 (unknown):
```unknown
Epoch 378/500:  76%|███████▌  | 378/500 [43:16<13:48,  6.79s/it, loss=1.47e+04, v_num=1]
```

Example 1196 (unknown):
```unknown
Epoch 379/500:  76%|███████▌  | 378/500 [43:16<13:48,  6.79s/it, loss=1.47e+04, v_num=1]
```

Example 1197 (unknown):
```unknown
Epoch 379/500:  76%|███████▌  | 379/500 [43:23<13:42,  6.80s/it, loss=1.47e+04, v_num=1]
```

Example 1198 (unknown):
```unknown
Epoch 379/500:  76%|███████▌  | 379/500 [43:23<13:42,  6.80s/it, loss=1.51e+04, v_num=1]
```

Example 1199 (unknown):
```unknown
Epoch 380/500:  76%|███████▌  | 379/500 [43:23<13:42,  6.80s/it, loss=1.51e+04, v_num=1]
```

Example 1200 (unknown):
```unknown
Epoch 380/500:  76%|███████▌  | 380/500 [43:30<13:34,  6.79s/it, loss=1.51e+04, v_num=1]
```

Example 1201 (unknown):
```unknown
Epoch 380/500:  76%|███████▌  | 380/500 [43:30<13:34,  6.79s/it, loss=1.46e+04, v_num=1]
```

Example 1202 (unknown):
```unknown
Epoch 381/500:  76%|███████▌  | 380/500 [43:30<13:34,  6.79s/it, loss=1.46e+04, v_num=1]
```

Example 1203 (unknown):
```unknown
Epoch 381/500:  76%|███████▌  | 381/500 [43:37<13:33,  6.84s/it, loss=1.46e+04, v_num=1]
```

Example 1204 (unknown):
```unknown
Epoch 381/500:  76%|███████▌  | 381/500 [43:37<13:33,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 1205 (unknown):
```unknown
Epoch 382/500:  76%|███████▌  | 381/500 [43:37<13:33,  6.84s/it, loss=1.49e+04, v_num=1]
```

Example 1206 (unknown):
```unknown
Epoch 382/500:  76%|███████▋  | 382/500 [43:43<13:18,  6.77s/it, loss=1.49e+04, v_num=1]
```

Example 1207 (unknown):
```unknown
Epoch 382/500:  76%|███████▋  | 382/500 [43:43<13:18,  6.77s/it, loss=1.45e+04, v_num=1]
```

Example 1208 (unknown):
```unknown
Epoch 383/500:  76%|███████▋  | 382/500 [43:43<13:18,  6.77s/it, loss=1.45e+04, v_num=1]
```

Example 1209 (unknown):
```unknown
Epoch 383/500:  77%|███████▋  | 383/500 [43:50<13:22,  6.86s/it, loss=1.45e+04, v_num=1]
```

Example 1210 (unknown):
```unknown
Epoch 383/500:  77%|███████▋  | 383/500 [43:50<13:22,  6.86s/it, loss=1.49e+04, v_num=1]
```

Example 1211 (unknown):
```unknown
Epoch 384/500:  77%|███████▋  | 383/500 [43:50<13:22,  6.86s/it, loss=1.49e+04, v_num=1]
```

Example 1212 (unknown):
```unknown
Epoch 384/500:  77%|███████▋  | 384/500 [43:57<13:22,  6.92s/it, loss=1.49e+04, v_num=1]
```

Example 1213 (unknown):
```unknown
Epoch 384/500:  77%|███████▋  | 384/500 [43:57<13:22,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1214 (unknown):
```unknown
Epoch 385/500:  77%|███████▋  | 384/500 [43:57<13:22,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1215 (unknown):
```unknown
Epoch 385/500:  77%|███████▋  | 385/500 [44:04<13:16,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1216 (unknown):
```unknown
Epoch 385/500:  77%|███████▋  | 385/500 [44:04<13:16,  6.92s/it, loss=1.52e+04, v_num=1]
```

Example 1217 (unknown):
```unknown
Epoch 386/500:  77%|███████▋  | 385/500 [44:04<13:16,  6.92s/it, loss=1.52e+04, v_num=1]
```

Example 1218 (unknown):
```unknown
Epoch 386/500:  77%|███████▋  | 386/500 [44:11<13:11,  6.94s/it, loss=1.52e+04, v_num=1]
```

Example 1219 (unknown):
```unknown
Epoch 386/500:  77%|███████▋  | 386/500 [44:11<13:11,  6.94s/it, loss=1.47e+04, v_num=1]
```

Example 1220 (unknown):
```unknown
Epoch 387/500:  77%|███████▋  | 386/500 [44:11<13:11,  6.94s/it, loss=1.47e+04, v_num=1]
```

Example 1221 (unknown):
```unknown
Epoch 387/500:  77%|███████▋  | 387/500 [44:18<13:07,  6.97s/it, loss=1.47e+04, v_num=1]
```

Example 1222 (unknown):
```unknown
Epoch 387/500:  77%|███████▋  | 387/500 [44:18<13:07,  6.97s/it, loss=1.47e+04, v_num=1]
```

Example 1223 (unknown):
```unknown
Epoch 388/500:  77%|███████▋  | 387/500 [44:18<13:07,  6.97s/it, loss=1.47e+04, v_num=1]
```

Example 1224 (unknown):
```unknown
Epoch 388/500:  78%|███████▊  | 388/500 [44:25<12:59,  6.96s/it, loss=1.47e+04, v_num=1]
```

Example 1225 (unknown):
```unknown
Epoch 388/500:  78%|███████▊  | 388/500 [44:25<12:59,  6.96s/it, loss=1.5e+04, v_num=1]
```

Example 1226 (unknown):
```unknown
Epoch 389/500:  78%|███████▊  | 388/500 [44:25<12:59,  6.96s/it, loss=1.5e+04, v_num=1]
```

Example 1227 (unknown):
```unknown
Epoch 389/500:  78%|███████▊  | 389/500 [44:32<12:52,  6.96s/it, loss=1.5e+04, v_num=1]
```

Example 1228 (unknown):
```unknown
Epoch 389/500:  78%|███████▊  | 389/500 [44:32<12:52,  6.96s/it, loss=1.47e+04, v_num=1]
```

Example 1229 (unknown):
```unknown
Epoch 390/500:  78%|███████▊  | 389/500 [44:32<12:52,  6.96s/it, loss=1.47e+04, v_num=1]
```

Example 1230 (unknown):
```unknown
Epoch 390/500:  78%|███████▊  | 390/500 [44:39<12:42,  6.93s/it, loss=1.47e+04, v_num=1]
```

Example 1231 (unknown):
```unknown
Epoch 390/500:  78%|███████▊  | 390/500 [44:39<12:42,  6.93s/it, loss=1.48e+04, v_num=1]
```

Example 1232 (unknown):
```unknown
Epoch 391/500:  78%|███████▊  | 390/500 [44:39<12:42,  6.93s/it, loss=1.48e+04, v_num=1]
```

Example 1233 (unknown):
```unknown
Epoch 391/500:  78%|███████▊  | 391/500 [44:46<12:32,  6.90s/it, loss=1.48e+04, v_num=1]
```

Example 1234 (unknown):
```unknown
Epoch 391/500:  78%|███████▊  | 391/500 [44:46<12:32,  6.90s/it, loss=1.48e+04, v_num=1]
```

Example 1235 (unknown):
```unknown
Epoch 392/500:  78%|███████▊  | 391/500 [44:46<12:32,  6.90s/it, loss=1.48e+04, v_num=1]
```

Example 1236 (unknown):
```unknown
Epoch 392/500:  78%|███████▊  | 392/500 [44:53<12:20,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 1237 (unknown):
```unknown
Epoch 392/500:  78%|███████▊  | 392/500 [44:53<12:20,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 1238 (unknown):
```unknown
Epoch 393/500:  78%|███████▊  | 392/500 [44:53<12:20,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 1239 (unknown):
```unknown
Epoch 393/500:  79%|███████▊  | 393/500 [45:00<12:10,  6.82s/it, loss=1.48e+04, v_num=1]
```

Example 1240 (unknown):
```unknown
Epoch 393/500:  79%|███████▊  | 393/500 [45:00<12:10,  6.82s/it, loss=1.45e+04, v_num=1]
```

Example 1241 (unknown):
```unknown
Epoch 394/500:  79%|███████▊  | 393/500 [45:00<12:10,  6.82s/it, loss=1.45e+04, v_num=1]
```

Example 1242 (unknown):
```unknown
Epoch 394/500:  79%|███████▉  | 394/500 [45:06<12:07,  6.86s/it, loss=1.45e+04, v_num=1]
```

Example 1243 (unknown):
```unknown
Epoch 394/500:  79%|███████▉  | 394/500 [45:06<12:07,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1244 (unknown):
```unknown
Epoch 395/500:  79%|███████▉  | 394/500 [45:06<12:07,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1245 (unknown):
```unknown
Epoch 395/500:  79%|███████▉  | 395/500 [45:13<12:05,  6.91s/it, loss=1.47e+04, v_num=1]
```

Example 1246 (unknown):
```unknown
Epoch 395/500:  79%|███████▉  | 395/500 [45:13<12:05,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 1247 (unknown):
```unknown
Epoch 396/500:  79%|███████▉  | 395/500 [45:13<12:05,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 1248 (unknown):
```unknown
Epoch 396/500:  79%|███████▉  | 396/500 [45:20<12:01,  6.93s/it, loss=1.5e+04, v_num=1]
```

Example 1249 (unknown):
```unknown
Epoch 396/500:  79%|███████▉  | 396/500 [45:20<12:01,  6.93s/it, loss=1.47e+04, v_num=1]
```

Example 1250 (unknown):
```unknown
Epoch 397/500:  79%|███████▉  | 396/500 [45:20<12:01,  6.93s/it, loss=1.47e+04, v_num=1]
```

Example 1251 (unknown):
```unknown
Epoch 397/500:  79%|███████▉  | 397/500 [45:27<11:49,  6.88s/it, loss=1.47e+04, v_num=1]
```

Example 1252 (unknown):
```unknown
Epoch 397/500:  79%|███████▉  | 397/500 [45:27<11:49,  6.88s/it, loss=1.46e+04, v_num=1]
```

Example 1253 (unknown):
```unknown
Epoch 398/500:  79%|███████▉  | 397/500 [45:27<11:49,  6.88s/it, loss=1.46e+04, v_num=1]
```

Example 1254 (unknown):
```unknown
Epoch 398/500:  80%|███████▉  | 398/500 [45:34<11:47,  6.94s/it, loss=1.46e+04, v_num=1]
```

Example 1255 (unknown):
```unknown
Epoch 398/500:  80%|███████▉  | 398/500 [45:34<11:47,  6.94s/it, loss=1.48e+04, v_num=1]
```

Example 1256 (unknown):
```unknown
Epoch 399/500:  80%|███████▉  | 398/500 [45:34<11:47,  6.94s/it, loss=1.48e+04, v_num=1]
```

Example 1257 (unknown):
```unknown
Epoch 399/500:  80%|███████▉  | 399/500 [45:41<11:43,  6.97s/it, loss=1.48e+04, v_num=1]
```

Example 1258 (unknown):
```unknown
Epoch 399/500:  80%|███████▉  | 399/500 [45:41<11:43,  6.97s/it, loss=1.49e+04, v_num=1]
```

Example 1259 (unknown):
```unknown
Epoch 400/500:  80%|███████▉  | 399/500 [45:41<11:43,  6.97s/it, loss=1.49e+04, v_num=1]
```

Example 1260 (unknown):
```unknown
Epoch 400/500:  80%|████████  | 400/500 [45:48<11:31,  6.91s/it, loss=1.49e+04, v_num=1]
```

Example 1261 (unknown):
```unknown
Epoch 400/500:  80%|████████  | 400/500 [45:48<11:31,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 1262 (unknown):
```unknown
Epoch 401/500:  80%|████████  | 400/500 [45:48<11:31,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 1263 (unknown):
```unknown
Epoch 401/500:  80%|████████  | 401/500 [45:55<11:24,  6.91s/it, loss=1.5e+04, v_num=1]
```

Example 1264 (unknown):
```unknown
Epoch 401/500:  80%|████████  | 401/500 [45:55<11:24,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1265 (unknown):
```unknown
Epoch 402/500:  80%|████████  | 401/500 [45:55<11:24,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1266 (unknown):
```unknown
Epoch 402/500:  80%|████████  | 402/500 [46:02<11:14,  6.88s/it, loss=1.48e+04, v_num=1]
```

Example 1267 (unknown):
```unknown
Epoch 402/500:  80%|████████  | 402/500 [46:02<11:14,  6.88s/it, loss=1.45e+04, v_num=1]
```

Example 1268 (unknown):
```unknown
Epoch 403/500:  80%|████████  | 402/500 [46:02<11:14,  6.88s/it, loss=1.45e+04, v_num=1]
```

Example 1269 (unknown):
```unknown
Epoch 403/500:  81%|████████  | 403/500 [46:09<11:10,  6.91s/it, loss=1.45e+04, v_num=1]
```

Example 1270 (unknown):
```unknown
Epoch 403/500:  81%|████████  | 403/500 [46:09<11:10,  6.91s/it, loss=1.47e+04, v_num=1]
```

Example 1271 (unknown):
```unknown
Epoch 404/500:  81%|████████  | 403/500 [46:09<11:10,  6.91s/it, loss=1.47e+04, v_num=1]
```

Example 1272 (unknown):
```unknown
Epoch 404/500:  81%|████████  | 404/500 [46:16<11:04,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1273 (unknown):
```unknown
Epoch 404/500:  81%|████████  | 404/500 [46:16<11:04,  6.92s/it, loss=1.45e+04, v_num=1]
```

Example 1274 (unknown):
```unknown
Epoch 405/500:  81%|████████  | 404/500 [46:16<11:04,  6.92s/it, loss=1.45e+04, v_num=1]
```

Example 1275 (unknown):
```unknown
Epoch 405/500:  81%|████████  | 405/500 [46:23<10:57,  6.92s/it, loss=1.45e+04, v_num=1]
```

Example 1276 (unknown):
```unknown
Epoch 405/500:  81%|████████  | 405/500 [46:23<10:57,  6.92s/it, loss=1.46e+04, v_num=1]
```

Example 1277 (unknown):
```unknown
Epoch 406/500:  81%|████████  | 405/500 [46:23<10:57,  6.92s/it, loss=1.46e+04, v_num=1]
```

Example 1278 (unknown):
```unknown
Epoch 406/500:  81%|████████  | 406/500 [46:30<10:51,  6.93s/it, loss=1.46e+04, v_num=1]
```

Example 1279 (unknown):
```unknown
Epoch 406/500:  81%|████████  | 406/500 [46:30<10:51,  6.93s/it, loss=1.46e+04, v_num=1]
```

Example 1280 (unknown):
```unknown
Epoch 407/500:  81%|████████  | 406/500 [46:30<10:51,  6.93s/it, loss=1.46e+04, v_num=1]
```

Example 1281 (unknown):
```unknown
Epoch 407/500:  81%|████████▏ | 407/500 [46:37<10:43,  6.92s/it, loss=1.46e+04, v_num=1]
```

Example 1282 (unknown):
```unknown
Epoch 407/500:  81%|████████▏ | 407/500 [46:37<10:43,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 1283 (unknown):
```unknown
Epoch 408/500:  81%|████████▏ | 407/500 [46:37<10:43,  6.92s/it, loss=1.51e+04, v_num=1]
```

Example 1284 (unknown):
```unknown
Epoch 408/500:  82%|████████▏ | 408/500 [46:44<10:38,  6.94s/it, loss=1.51e+04, v_num=1]
```

Example 1285 (unknown):
```unknown
Epoch 408/500:  82%|████████▏ | 408/500 [46:44<10:38,  6.94s/it, loss=1.5e+04, v_num=1]
```

Example 1286 (unknown):
```unknown
Epoch 409/500:  82%|████████▏ | 408/500 [46:44<10:38,  6.94s/it, loss=1.5e+04, v_num=1]
```

Example 1287 (unknown):
```unknown
Epoch 409/500:  82%|████████▏ | 409/500 [46:51<10:33,  6.96s/it, loss=1.5e+04, v_num=1]
```

Example 1288 (unknown):
```unknown
Epoch 409/500:  82%|████████▏ | 409/500 [46:51<10:33,  6.96s/it, loss=1.48e+04, v_num=1]
```

Example 1289 (unknown):
```unknown
Epoch 410/500:  82%|████████▏ | 409/500 [46:51<10:33,  6.96s/it, loss=1.48e+04, v_num=1]
```

Example 1290 (unknown):
```unknown
Epoch 410/500:  82%|████████▏ | 410/500 [46:57<10:22,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1291 (unknown):
```unknown
Epoch 410/500:  82%|████████▏ | 410/500 [46:57<10:22,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1292 (unknown):
```unknown
Epoch 411/500:  82%|████████▏ | 410/500 [46:57<10:22,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1293 (unknown):
```unknown
Epoch 411/500:  82%|████████▏ | 411/500 [47:04<10:07,  6.82s/it, loss=1.48e+04, v_num=1]
```

Example 1294 (unknown):
```unknown
Epoch 411/500:  82%|████████▏ | 411/500 [47:04<10:07,  6.82s/it, loss=1.48e+04, v_num=1]
```

Example 1295 (unknown):
```unknown
Epoch 412/500:  82%|████████▏ | 411/500 [47:04<10:07,  6.82s/it, loss=1.48e+04, v_num=1]
```

Example 1296 (unknown):
```unknown
Epoch 412/500:  82%|████████▏ | 412/500 [47:11<10:03,  6.86s/it, loss=1.48e+04, v_num=1]
```

Example 1297 (unknown):
```unknown
Epoch 412/500:  82%|████████▏ | 412/500 [47:11<10:03,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1298 (unknown):
```unknown
Epoch 413/500:  82%|████████▏ | 412/500 [47:11<10:03,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1299 (unknown):
```unknown
Epoch 413/500:  83%|████████▎ | 413/500 [47:18<09:55,  6.85s/it, loss=1.47e+04, v_num=1]
```

Example 1300 (unknown):
```unknown
Epoch 413/500:  83%|████████▎ | 413/500 [47:18<09:55,  6.85s/it, loss=1.47e+04, v_num=1]
```

Example 1301 (unknown):
```unknown
Epoch 414/500:  83%|████████▎ | 413/500 [47:18<09:55,  6.85s/it, loss=1.47e+04, v_num=1]
```

Example 1302 (unknown):
```unknown
Epoch 414/500:  83%|████████▎ | 414/500 [47:24<09:42,  6.77s/it, loss=1.47e+04, v_num=1]
```

Example 1303 (unknown):
```unknown
Epoch 414/500:  83%|████████▎ | 414/500 [47:24<09:42,  6.77s/it, loss=1.47e+04, v_num=1]
```

Example 1304 (unknown):
```unknown
Epoch 415/500:  83%|████████▎ | 414/500 [47:24<09:42,  6.77s/it, loss=1.47e+04, v_num=1]
```

Example 1305 (unknown):
```unknown
Epoch 415/500:  83%|████████▎ | 415/500 [47:31<09:40,  6.83s/it, loss=1.47e+04, v_num=1]
```

Example 1306 (unknown):
```unknown
Epoch 415/500:  83%|████████▎ | 415/500 [47:31<09:40,  6.83s/it, loss=1.47e+04, v_num=1]
```

Example 1307 (unknown):
```unknown
Epoch 416/500:  83%|████████▎ | 415/500 [47:31<09:40,  6.83s/it, loss=1.47e+04, v_num=1]
```

Example 1308 (unknown):
```unknown
Epoch 416/500:  83%|████████▎ | 416/500 [47:38<09:37,  6.88s/it, loss=1.47e+04, v_num=1]
```

Example 1309 (unknown):
```unknown
Epoch 416/500:  83%|████████▎ | 416/500 [47:38<09:37,  6.88s/it, loss=1.47e+04, v_num=1]
```

Example 1310 (unknown):
```unknown
Epoch 417/500:  83%|████████▎ | 416/500 [47:38<09:37,  6.88s/it, loss=1.47e+04, v_num=1]
```

Example 1311 (unknown):
```unknown
Epoch 417/500:  83%|████████▎ | 417/500 [47:45<09:36,  6.94s/it, loss=1.47e+04, v_num=1]
```

Example 1312 (unknown):
```unknown
Epoch 417/500:  83%|████████▎ | 417/500 [47:45<09:36,  6.94s/it, loss=1.49e+04, v_num=1]
```

Example 1313 (unknown):
```unknown
Epoch 418/500:  83%|████████▎ | 417/500 [47:45<09:36,  6.94s/it, loss=1.49e+04, v_num=1]
```

Example 1314 (unknown):
```unknown
Epoch 418/500:  84%|████████▎ | 418/500 [47:52<09:29,  6.94s/it, loss=1.49e+04, v_num=1]
```

Example 1315 (unknown):
```unknown
Epoch 418/500:  84%|████████▎ | 418/500 [47:52<09:29,  6.94s/it, loss=1.48e+04, v_num=1]
```

Example 1316 (unknown):
```unknown
Epoch 419/500:  84%|████████▎ | 418/500 [47:52<09:29,  6.94s/it, loss=1.48e+04, v_num=1]
```

Example 1317 (unknown):
```unknown
Epoch 419/500:  84%|████████▍ | 419/500 [47:59<09:19,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1318 (unknown):
```unknown
Epoch 419/500:  84%|████████▍ | 419/500 [47:59<09:19,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1319 (unknown):
```unknown
Epoch 420/500:  84%|████████▍ | 419/500 [47:59<09:19,  6.91s/it, loss=1.48e+04, v_num=1]
```

Example 1320 (unknown):
```unknown
Epoch 420/500:  84%|████████▍ | 420/500 [48:06<09:15,  6.94s/it, loss=1.48e+04, v_num=1]
```

Example 1321 (unknown):
```unknown
Epoch 420/500:  84%|████████▍ | 420/500 [48:06<09:15,  6.94s/it, loss=1.46e+04, v_num=1]
```

Example 1322 (unknown):
```unknown
Epoch 421/500:  84%|████████▍ | 420/500 [48:06<09:15,  6.94s/it, loss=1.46e+04, v_num=1]
```

Example 1323 (unknown):
```unknown
Epoch 421/500:  84%|████████▍ | 421/500 [48:13<09:08,  6.95s/it, loss=1.46e+04, v_num=1]
```

Example 1324 (unknown):
```unknown
Epoch 421/500:  84%|████████▍ | 421/500 [48:13<09:08,  6.95s/it, loss=1.46e+04, v_num=1]
```

Example 1325 (unknown):
```unknown
Epoch 422/500:  84%|████████▍ | 421/500 [48:13<09:08,  6.95s/it, loss=1.46e+04, v_num=1]
```

Example 1326 (unknown):
```unknown
Epoch 422/500:  84%|████████▍ | 422/500 [48:20<08:59,  6.92s/it, loss=1.46e+04, v_num=1]
```

Example 1327 (unknown):
```unknown
Epoch 422/500:  84%|████████▍ | 422/500 [48:20<08:59,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1328 (unknown):
```unknown
Epoch 423/500:  84%|████████▍ | 422/500 [48:20<08:59,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1329 (unknown):
```unknown
Epoch 423/500:  85%|████████▍ | 423/500 [48:27<08:52,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1330 (unknown):
```unknown
Epoch 423/500:  85%|████████▍ | 423/500 [48:27<08:52,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1331 (unknown):
```unknown
Epoch 424/500:  85%|████████▍ | 423/500 [48:27<08:52,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1332 (unknown):
```unknown
Epoch 424/500:  85%|████████▍ | 424/500 [48:34<08:47,  6.93s/it, loss=1.47e+04, v_num=1]
```

Example 1333 (unknown):
```unknown
Epoch 424/500:  85%|████████▍ | 424/500 [48:34<08:47,  6.93s/it, loss=1.46e+04, v_num=1]
```

Example 1334 (unknown):
```unknown
Epoch 425/500:  85%|████████▍ | 424/500 [48:34<08:47,  6.93s/it, loss=1.46e+04, v_num=1]
```

Example 1335 (unknown):
```unknown
Epoch 425/500:  85%|████████▌ | 425/500 [48:41<08:41,  6.95s/it, loss=1.46e+04, v_num=1]
```

Example 1336 (unknown):
```unknown
Epoch 425/500:  85%|████████▌ | 425/500 [48:41<08:41,  6.95s/it, loss=1.41e+04, v_num=1]
```

Example 1337 (unknown):
```unknown
Epoch 426/500:  85%|████████▌ | 425/500 [48:41<08:41,  6.95s/it, loss=1.41e+04, v_num=1]
```

Example 1338 (unknown):
```unknown
Epoch 426/500:  85%|████████▌ | 426/500 [48:48<08:34,  6.95s/it, loss=1.41e+04, v_num=1]
```

Example 1339 (unknown):
```unknown
Epoch 426/500:  85%|████████▌ | 426/500 [48:48<08:34,  6.95s/it, loss=1.43e+04, v_num=1]
```

Example 1340 (unknown):
```unknown
Epoch 427/500:  85%|████████▌ | 426/500 [48:48<08:34,  6.95s/it, loss=1.43e+04, v_num=1]
```

Example 1341 (unknown):
```unknown
Epoch 427/500:  85%|████████▌ | 427/500 [48:55<08:25,  6.93s/it, loss=1.43e+04, v_num=1]
```

Example 1342 (unknown):
```unknown
Epoch 427/500:  85%|████████▌ | 427/500 [48:55<08:25,  6.93s/it, loss=1.49e+04, v_num=1]
```

Example 1343 (unknown):
```unknown
Epoch 428/500:  85%|████████▌ | 427/500 [48:55<08:25,  6.93s/it, loss=1.49e+04, v_num=1]
```

Example 1344 (unknown):
```unknown
Epoch 428/500:  86%|████████▌ | 428/500 [49:01<08:11,  6.83s/it, loss=1.49e+04, v_num=1]
```

Example 1345 (unknown):
```unknown
Epoch 428/500:  86%|████████▌ | 428/500 [49:01<08:11,  6.83s/it, loss=1.44e+04, v_num=1]
```

Example 1346 (unknown):
```unknown
Epoch 429/500:  86%|████████▌ | 428/500 [49:01<08:11,  6.83s/it, loss=1.44e+04, v_num=1]
```

Example 1347 (unknown):
```unknown
Epoch 429/500:  86%|████████▌ | 429/500 [49:08<07:59,  6.75s/it, loss=1.44e+04, v_num=1]
```

Example 1348 (unknown):
```unknown
Epoch 429/500:  86%|████████▌ | 429/500 [49:08<07:59,  6.75s/it, loss=1.47e+04, v_num=1]
```

Example 1349 (unknown):
```unknown
Epoch 430/500:  86%|████████▌ | 429/500 [49:08<07:59,  6.75s/it, loss=1.47e+04, v_num=1]
```

Example 1350 (unknown):
```unknown
Epoch 430/500:  86%|████████▌ | 430/500 [49:15<07:56,  6.80s/it, loss=1.47e+04, v_num=1]
```

Example 1351 (unknown):
```unknown
Epoch 430/500:  86%|████████▌ | 430/500 [49:15<07:56,  6.80s/it, loss=1.47e+04, v_num=1]
```

Example 1352 (unknown):
```unknown
Epoch 431/500:  86%|████████▌ | 430/500 [49:15<07:56,  6.80s/it, loss=1.47e+04, v_num=1]
```

Example 1353 (unknown):
```unknown
Epoch 431/500:  86%|████████▌ | 431/500 [49:22<07:52,  6.85s/it, loss=1.47e+04, v_num=1]
```

Example 1354 (unknown):
```unknown
Epoch 431/500:  86%|████████▌ | 431/500 [49:22<07:52,  6.85s/it, loss=1.46e+04, v_num=1]
```

Example 1355 (unknown):
```unknown
Epoch 432/500:  86%|████████▌ | 431/500 [49:22<07:52,  6.85s/it, loss=1.46e+04, v_num=1]
```

Example 1356 (unknown):
```unknown
Epoch 432/500:  86%|████████▋ | 432/500 [49:29<07:46,  6.86s/it, loss=1.46e+04, v_num=1]
```

Example 1357 (unknown):
```unknown
Epoch 432/500:  86%|████████▋ | 432/500 [49:29<07:46,  6.86s/it, loss=1.46e+04, v_num=1]
```

Example 1358 (unknown):
```unknown
Epoch 433/500:  86%|████████▋ | 432/500 [49:29<07:46,  6.86s/it, loss=1.46e+04, v_num=1]
```

Example 1359 (unknown):
```unknown
Epoch 433/500:  87%|████████▋ | 433/500 [49:36<07:41,  6.89s/it, loss=1.46e+04, v_num=1]
```

Example 1360 (unknown):
```unknown
Epoch 433/500:  87%|████████▋ | 433/500 [49:36<07:41,  6.89s/it, loss=1.44e+04, v_num=1]
```

Example 1361 (unknown):
```unknown
Epoch 434/500:  87%|████████▋ | 433/500 [49:36<07:41,  6.89s/it, loss=1.44e+04, v_num=1]
```

Example 1362 (unknown):
```unknown
Epoch 434/500:  87%|████████▋ | 434/500 [49:42<07:25,  6.75s/it, loss=1.44e+04, v_num=1]
```

Example 1363 (unknown):
```unknown
Epoch 434/500:  87%|████████▋ | 434/500 [49:42<07:25,  6.75s/it, loss=1.47e+04, v_num=1]
```

Example 1364 (unknown):
```unknown
Epoch 435/500:  87%|████████▋ | 434/500 [49:42<07:25,  6.75s/it, loss=1.47e+04, v_num=1]
```

Example 1365 (unknown):
```unknown
Epoch 435/500:  87%|████████▋ | 435/500 [49:49<07:19,  6.76s/it, loss=1.47e+04, v_num=1]
```

Example 1366 (unknown):
```unknown
Epoch 435/500:  87%|████████▋ | 435/500 [49:49<07:19,  6.76s/it, loss=1.45e+04, v_num=1]
```

Example 1367 (unknown):
```unknown
Epoch 436/500:  87%|████████▋ | 435/500 [49:49<07:19,  6.76s/it, loss=1.45e+04, v_num=1]
```

Example 1368 (unknown):
```unknown
Epoch 436/500:  87%|████████▋ | 436/500 [49:55<07:11,  6.75s/it, loss=1.45e+04, v_num=1]
```

Example 1369 (unknown):
```unknown
Epoch 436/500:  87%|████████▋ | 436/500 [49:55<07:11,  6.75s/it, loss=1.48e+04, v_num=1]
```

Example 1370 (unknown):
```unknown
Epoch 437/500:  87%|████████▋ | 436/500 [49:55<07:11,  6.75s/it, loss=1.48e+04, v_num=1]
```

Example 1371 (unknown):
```unknown
Epoch 437/500:  87%|████████▋ | 437/500 [50:02<07:04,  6.73s/it, loss=1.48e+04, v_num=1]
```

Example 1372 (unknown):
```unknown
Epoch 437/500:  87%|████████▋ | 437/500 [50:02<07:04,  6.73s/it, loss=1.49e+04, v_num=1]
```

Example 1373 (unknown):
```unknown
Epoch 438/500:  87%|████████▋ | 437/500 [50:02<07:04,  6.73s/it, loss=1.49e+04, v_num=1]
```

Example 1374 (unknown):
```unknown
Epoch 438/500:  88%|████████▊ | 438/500 [50:09<07:03,  6.82s/it, loss=1.49e+04, v_num=1]
```

Example 1375 (unknown):
```unknown
Epoch 438/500:  88%|████████▊ | 438/500 [50:09<07:03,  6.82s/it, loss=1.49e+04, v_num=1]
```

Example 1376 (unknown):
```unknown
Epoch 439/500:  88%|████████▊ | 438/500 [50:09<07:03,  6.82s/it, loss=1.49e+04, v_num=1]
```

Example 1377 (unknown):
```unknown
Epoch 439/500:  88%|████████▊ | 439/500 [50:16<06:53,  6.77s/it, loss=1.49e+04, v_num=1]
```

Example 1378 (unknown):
```unknown
Epoch 439/500:  88%|████████▊ | 439/500 [50:16<06:53,  6.77s/it, loss=1.47e+04, v_num=1]
```

Example 1379 (unknown):
```unknown
Epoch 440/500:  88%|████████▊ | 439/500 [50:16<06:53,  6.77s/it, loss=1.47e+04, v_num=1]
```

Example 1380 (unknown):
```unknown
Epoch 440/500:  88%|████████▊ | 440/500 [50:23<06:48,  6.80s/it, loss=1.47e+04, v_num=1]
```

Example 1381 (unknown):
```unknown
Epoch 440/500:  88%|████████▊ | 440/500 [50:23<06:48,  6.80s/it, loss=1.46e+04, v_num=1]
```

Example 1382 (unknown):
```unknown
Epoch 441/500:  88%|████████▊ | 440/500 [50:23<06:48,  6.80s/it, loss=1.46e+04, v_num=1]
```

Example 1383 (unknown):
```unknown
Epoch 441/500:  88%|████████▊ | 441/500 [50:30<06:44,  6.85s/it, loss=1.46e+04, v_num=1]
```

Example 1384 (unknown):
```unknown
Epoch 441/500:  88%|████████▊ | 441/500 [50:30<06:44,  6.85s/it, loss=1.47e+04, v_num=1]
```

Example 1385 (unknown):
```unknown
Epoch 442/500:  88%|████████▊ | 441/500 [50:30<06:44,  6.85s/it, loss=1.47e+04, v_num=1]
```

Example 1386 (unknown):
```unknown
Epoch 442/500:  88%|████████▊ | 442/500 [50:37<06:38,  6.87s/it, loss=1.47e+04, v_num=1]
```

Example 1387 (unknown):
```unknown
Epoch 442/500:  88%|████████▊ | 442/500 [50:37<06:38,  6.87s/it, loss=1.45e+04, v_num=1]
```

Example 1388 (unknown):
```unknown
Epoch 443/500:  88%|████████▊ | 442/500 [50:37<06:38,  6.87s/it, loss=1.45e+04, v_num=1]
```

Example 1389 (unknown):
```unknown
Epoch 443/500:  89%|████████▊ | 443/500 [50:43<06:31,  6.86s/it, loss=1.45e+04, v_num=1]
```

Example 1390 (unknown):
```unknown
Epoch 443/500:  89%|████████▊ | 443/500 [50:43<06:31,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1391 (unknown):
```unknown
Epoch 444/500:  89%|████████▊ | 443/500 [50:43<06:31,  6.86s/it, loss=1.47e+04, v_num=1]
```

Example 1392 (unknown):
```unknown
Epoch 444/500:  89%|████████▉ | 444/500 [50:50<06:21,  6.82s/it, loss=1.47e+04, v_num=1]
```

Example 1393 (unknown):
```unknown
Epoch 444/500:  89%|████████▉ | 444/500 [50:50<06:21,  6.82s/it, loss=1.46e+04, v_num=1]
```

Example 1394 (unknown):
```unknown
Epoch 445/500:  89%|████████▉ | 444/500 [50:50<06:21,  6.82s/it, loss=1.46e+04, v_num=1]
```

Example 1395 (unknown):
```unknown
Epoch 445/500:  89%|████████▉ | 445/500 [50:57<06:19,  6.90s/it, loss=1.46e+04, v_num=1]
```

Example 1396 (unknown):
```unknown
Epoch 445/500:  89%|████████▉ | 445/500 [50:57<06:19,  6.90s/it, loss=1.46e+04, v_num=1]
```

Example 1397 (unknown):
```unknown
Epoch 446/500:  89%|████████▉ | 445/500 [50:57<06:19,  6.90s/it, loss=1.46e+04, v_num=1]
```

Example 1398 (unknown):
```unknown
Epoch 446/500:  89%|████████▉ | 446/500 [51:04<06:13,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1399 (unknown):
```unknown
Epoch 446/500:  89%|████████▉ | 446/500 [51:04<06:13,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1400 (unknown):
```unknown
Epoch 447/500:  89%|████████▉ | 446/500 [51:04<06:13,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1401 (unknown):
```unknown
Epoch 447/500:  89%|████████▉ | 447/500 [51:11<06:05,  6.90s/it, loss=1.46e+04, v_num=1]
```

Example 1402 (unknown):
```unknown
Epoch 447/500:  89%|████████▉ | 447/500 [51:11<06:05,  6.90s/it, loss=1.45e+04, v_num=1]
```

Example 1403 (unknown):
```unknown
Epoch 448/500:  89%|████████▉ | 447/500 [51:11<06:05,  6.90s/it, loss=1.45e+04, v_num=1]
```

Example 1404 (unknown):
```unknown
Epoch 448/500:  90%|████████▉ | 448/500 [51:18<06:00,  6.93s/it, loss=1.45e+04, v_num=1]
```

Example 1405 (unknown):
```unknown
Epoch 448/500:  90%|████████▉ | 448/500 [51:18<06:00,  6.93s/it, loss=1.45e+04, v_num=1]
```

Example 1406 (unknown):
```unknown
Epoch 449/500:  90%|████████▉ | 448/500 [51:18<06:00,  6.93s/it, loss=1.45e+04, v_num=1]
```

Example 1407 (unknown):
```unknown
Epoch 449/500:  90%|████████▉ | 449/500 [51:25<05:52,  6.91s/it, loss=1.45e+04, v_num=1]
```

Example 1408 (unknown):
```unknown
Epoch 449/500:  90%|████████▉ | 449/500 [51:25<05:52,  6.91s/it, loss=1.49e+04, v_num=1]
```

Example 1409 (unknown):
```unknown
Epoch 450/500:  90%|████████▉ | 449/500 [51:25<05:52,  6.91s/it, loss=1.49e+04, v_num=1]
```

Example 1410 (unknown):
```unknown
Epoch 450/500:  90%|█████████ | 450/500 [51:32<05:46,  6.93s/it, loss=1.49e+04, v_num=1]
```

Example 1411 (unknown):
```unknown
Epoch 450/500:  90%|█████████ | 450/500 [51:32<05:46,  6.93s/it, loss=1.46e+04, v_num=1]
```

Example 1412 (unknown):
```unknown
Epoch 451/500:  90%|█████████ | 450/500 [51:32<05:46,  6.93s/it, loss=1.46e+04, v_num=1]
```

Example 1413 (unknown):
```unknown
Epoch 451/500:  90%|█████████ | 451/500 [51:39<05:37,  6.90s/it, loss=1.46e+04, v_num=1]
```

Example 1414 (unknown):
```unknown
Epoch 451/500:  90%|█████████ | 451/500 [51:39<05:37,  6.90s/it, loss=1.49e+04, v_num=1]
```

Example 1415 (unknown):
```unknown
Epoch 452/500:  90%|█████████ | 451/500 [51:39<05:37,  6.90s/it, loss=1.49e+04, v_num=1]
```

Example 1416 (unknown):
```unknown
Epoch 452/500:  90%|█████████ | 452/500 [51:45<05:28,  6.85s/it, loss=1.49e+04, v_num=1]
```

Example 1417 (unknown):
```unknown
Epoch 452/500:  90%|█████████ | 452/500 [51:45<05:28,  6.85s/it, loss=1.48e+04, v_num=1]
```

Example 1418 (unknown):
```unknown
Epoch 453/500:  90%|█████████ | 452/500 [51:45<05:28,  6.85s/it, loss=1.48e+04, v_num=1]
```

Example 1419 (unknown):
```unknown
Epoch 453/500:  91%|█████████ | 453/500 [51:52<05:23,  6.88s/it, loss=1.48e+04, v_num=1]
```

Example 1420 (unknown):
```unknown
Epoch 453/500:  91%|█████████ | 453/500 [51:52<05:23,  6.88s/it, loss=1.45e+04, v_num=1]
```

Example 1421 (unknown):
```unknown
Epoch 454/500:  91%|█████████ | 453/500 [51:52<05:23,  6.88s/it, loss=1.45e+04, v_num=1]
```

Example 1422 (unknown):
```unknown
Epoch 454/500:  91%|█████████ | 454/500 [51:59<05:17,  6.91s/it, loss=1.45e+04, v_num=1]
```

Example 1423 (unknown):
```unknown
Epoch 454/500:  91%|█████████ | 454/500 [51:59<05:17,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1424 (unknown):
```unknown
Epoch 455/500:  91%|█████████ | 454/500 [51:59<05:17,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1425 (unknown):
```unknown
Epoch 455/500:  91%|█████████ | 455/500 [52:06<05:11,  6.92s/it, loss=1.46e+04, v_num=1]
```

Example 1426 (unknown):
```unknown
Epoch 455/500:  91%|█████████ | 455/500 [52:06<05:11,  6.92s/it, loss=1.45e+04, v_num=1]
```

Example 1427 (unknown):
```unknown
Epoch 456/500:  91%|█████████ | 455/500 [52:06<05:11,  6.92s/it, loss=1.45e+04, v_num=1]
```

Example 1428 (unknown):
```unknown
Epoch 456/500:  91%|█████████ | 456/500 [52:13<05:04,  6.91s/it, loss=1.45e+04, v_num=1]
```

Example 1429 (unknown):
```unknown
Epoch 456/500:  91%|█████████ | 456/500 [52:13<05:04,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1430 (unknown):
```unknown
Epoch 457/500:  91%|█████████ | 456/500 [52:13<05:04,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1431 (unknown):
```unknown
Epoch 457/500:  91%|█████████▏| 457/500 [52:20<04:57,  6.92s/it, loss=1.46e+04, v_num=1]
```

Example 1432 (unknown):
```unknown
Epoch 457/500:  91%|█████████▏| 457/500 [52:20<04:57,  6.92s/it, loss=1.43e+04, v_num=1]
```

Example 1433 (unknown):
```unknown
Epoch 458/500:  91%|█████████▏| 457/500 [52:20<04:57,  6.92s/it, loss=1.43e+04, v_num=1]
```

Example 1434 (unknown):
```unknown
Epoch 458/500:  92%|█████████▏| 458/500 [52:27<04:50,  6.91s/it, loss=1.43e+04, v_num=1]
```

Example 1435 (unknown):
```unknown
Epoch 458/500:  92%|█████████▏| 458/500 [52:27<04:50,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1436 (unknown):
```unknown
Epoch 459/500:  92%|█████████▏| 458/500 [52:27<04:50,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1437 (unknown):
```unknown
Epoch 459/500:  92%|█████████▏| 459/500 [52:34<04:40,  6.85s/it, loss=1.46e+04, v_num=1]
```

Example 1438 (unknown):
```unknown
Epoch 459/500:  92%|█████████▏| 459/500 [52:34<04:40,  6.85s/it, loss=1.47e+04, v_num=1]
```

Example 1439 (unknown):
```unknown
Epoch 460/500:  92%|█████████▏| 459/500 [52:34<04:40,  6.85s/it, loss=1.47e+04, v_num=1]
```

Example 1440 (unknown):
```unknown
Epoch 460/500:  92%|█████████▏| 460/500 [52:40<04:31,  6.79s/it, loss=1.47e+04, v_num=1]
```

Example 1441 (unknown):
```unknown
Epoch 460/500:  92%|█████████▏| 460/500 [52:40<04:31,  6.79s/it, loss=1.44e+04, v_num=1]
```

Example 1442 (unknown):
```unknown
Epoch 461/500:  92%|█████████▏| 460/500 [52:40<04:31,  6.79s/it, loss=1.44e+04, v_num=1]
```

Example 1443 (unknown):
```unknown
Epoch 461/500:  92%|█████████▏| 461/500 [52:47<04:26,  6.83s/it, loss=1.44e+04, v_num=1]
```

Example 1444 (unknown):
```unknown
Epoch 461/500:  92%|█████████▏| 461/500 [52:47<04:26,  6.83s/it, loss=1.41e+04, v_num=1]
```

Example 1445 (unknown):
```unknown
Epoch 462/500:  92%|█████████▏| 461/500 [52:47<04:26,  6.83s/it, loss=1.41e+04, v_num=1]
```

Example 1446 (unknown):
```unknown
Epoch 462/500:  92%|█████████▏| 462/500 [52:54<04:22,  6.91s/it, loss=1.41e+04, v_num=1]
```

Example 1447 (unknown):
```unknown
Epoch 462/500:  92%|█████████▏| 462/500 [52:54<04:22,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1448 (unknown):
```unknown
Epoch 463/500:  92%|█████████▏| 462/500 [52:54<04:22,  6.91s/it, loss=1.46e+04, v_num=1]
```

Example 1449 (unknown):
```unknown
Epoch 463/500:  93%|█████████▎| 463/500 [53:01<04:16,  6.93s/it, loss=1.46e+04, v_num=1]
```

Example 1450 (unknown):
```unknown
Epoch 463/500:  93%|█████████▎| 463/500 [53:01<04:16,  6.93s/it, loss=1.47e+04, v_num=1]
```

Example 1451 (unknown):
```unknown
Epoch 464/500:  93%|█████████▎| 463/500 [53:01<04:16,  6.93s/it, loss=1.47e+04, v_num=1]
```

Example 1452 (unknown):
```unknown
Epoch 464/500:  93%|█████████▎| 464/500 [53:08<04:08,  6.92s/it, loss=1.47e+04, v_num=1]
```

Example 1453 (unknown):
```unknown
Epoch 464/500:  93%|█████████▎| 464/500 [53:08<04:08,  6.92s/it, loss=1.44e+04, v_num=1]
```

Example 1454 (unknown):
```unknown
Epoch 465/500:  93%|█████████▎| 464/500 [53:08<04:08,  6.92s/it, loss=1.44e+04, v_num=1]
```

Example 1455 (unknown):
```unknown
Epoch 465/500:  93%|█████████▎| 465/500 [53:15<04:03,  6.95s/it, loss=1.44e+04, v_num=1]
```

Example 1456 (unknown):
```unknown
Epoch 465/500:  93%|█████████▎| 465/500 [53:15<04:03,  6.95s/it, loss=1.46e+04, v_num=1]
```

Example 1457 (unknown):
```unknown
Epoch 466/500:  93%|█████████▎| 465/500 [53:15<04:03,  6.95s/it, loss=1.46e+04, v_num=1]
```

Example 1458 (unknown):
```unknown
Epoch 466/500:  93%|█████████▎| 466/500 [53:22<03:56,  6.95s/it, loss=1.46e+04, v_num=1]
```

Example 1459 (unknown):
```unknown
Epoch 466/500:  93%|█████████▎| 466/500 [53:22<03:56,  6.95s/it, loss=1.46e+04, v_num=1]
```

Example 1460 (unknown):
```unknown
Epoch 467/500:  93%|█████████▎| 466/500 [53:22<03:56,  6.95s/it, loss=1.46e+04, v_num=1]
```

Example 1461 (unknown):
```unknown
Epoch 467/500:  93%|█████████▎| 467/500 [53:29<03:48,  6.93s/it, loss=1.46e+04, v_num=1]
```

Example 1462 (unknown):
```unknown
Epoch 467/500:  93%|█████████▎| 467/500 [53:29<03:48,  6.93s/it, loss=1.45e+04, v_num=1]
```

Example 1463 (unknown):
```unknown
Epoch 468/500:  93%|█████████▎| 467/500 [53:29<03:48,  6.93s/it, loss=1.45e+04, v_num=1]
```

Example 1464 (unknown):
```unknown
Epoch 468/500:  94%|█████████▎| 468/500 [53:36<03:42,  6.94s/it, loss=1.45e+04, v_num=1]
```

Example 1465 (unknown):
```unknown
Epoch 468/500:  94%|█████████▎| 468/500 [53:36<03:42,  6.94s/it, loss=1.44e+04, v_num=1]
```

Example 1466 (unknown):
```unknown
Epoch 469/500:  94%|█████████▎| 468/500 [53:36<03:42,  6.94s/it, loss=1.44e+04, v_num=1]
```

Example 1467 (unknown):
```unknown
Epoch 469/500:  94%|█████████▍| 469/500 [53:43<03:32,  6.84s/it, loss=1.44e+04, v_num=1]
```

Example 1468 (unknown):
```unknown
Epoch 469/500:  94%|█████████▍| 469/500 [53:43<03:32,  6.84s/it, loss=1.47e+04, v_num=1]
```

Example 1469 (unknown):
```unknown
Epoch 470/500:  94%|█████████▍| 469/500 [53:43<03:32,  6.84s/it, loss=1.47e+04, v_num=1]
```

Example 1470 (unknown):
```unknown
Epoch 470/500:  94%|█████████▍| 470/500 [53:50<03:25,  6.84s/it, loss=1.47e+04, v_num=1]
```

Example 1471 (unknown):
```unknown
Epoch 470/500:  94%|█████████▍| 470/500 [53:50<03:25,  6.84s/it, loss=1.47e+04, v_num=1]
```

Example 1472 (unknown):
```unknown
Epoch 471/500:  94%|█████████▍| 470/500 [53:50<03:25,  6.84s/it, loss=1.47e+04, v_num=1]
```

Example 1473 (unknown):
```unknown
Epoch 471/500:  94%|█████████▍| 471/500 [53:56<03:17,  6.81s/it, loss=1.47e+04, v_num=1]
```

Example 1474 (unknown):
```unknown
Epoch 471/500:  94%|█████████▍| 471/500 [53:56<03:17,  6.81s/it, loss=1.47e+04, v_num=1]
```

Example 1475 (unknown):
```unknown
Epoch 472/500:  94%|█████████▍| 471/500 [53:56<03:17,  6.81s/it, loss=1.47e+04, v_num=1]
```

Example 1476 (unknown):
```unknown
Epoch 472/500:  94%|█████████▍| 472/500 [54:03<03:10,  6.79s/it, loss=1.47e+04, v_num=1]
```

Example 1477 (unknown):
```unknown
Epoch 472/500:  94%|█████████▍| 472/500 [54:03<03:10,  6.79s/it, loss=1.48e+04, v_num=1]
```

Example 1478 (unknown):
```unknown
Epoch 473/500:  94%|█████████▍| 472/500 [54:03<03:10,  6.79s/it, loss=1.48e+04, v_num=1]
```

Example 1479 (unknown):
```unknown
Epoch 473/500:  95%|█████████▍| 473/500 [54:10<03:04,  6.84s/it, loss=1.48e+04, v_num=1]
```

Example 1480 (unknown):
```unknown
Epoch 473/500:  95%|█████████▍| 473/500 [54:10<03:04,  6.84s/it, loss=1.45e+04, v_num=1]
```

Example 1481 (unknown):
```unknown
Epoch 474/500:  95%|█████████▍| 473/500 [54:10<03:04,  6.84s/it, loss=1.45e+04, v_num=1]
```

Example 1482 (unknown):
```unknown
Epoch 474/500:  95%|█████████▍| 474/500 [54:17<02:58,  6.85s/it, loss=1.45e+04, v_num=1]
```

Example 1483 (unknown):
```unknown
Epoch 474/500:  95%|█████████▍| 474/500 [54:17<02:58,  6.85s/it, loss=1.44e+04, v_num=1]
```

Example 1484 (unknown):
```unknown
Epoch 475/500:  95%|█████████▍| 474/500 [54:17<02:58,  6.85s/it, loss=1.44e+04, v_num=1]
```

Example 1485 (unknown):
```unknown
Epoch 475/500:  95%|█████████▌| 475/500 [54:24<02:51,  6.85s/it, loss=1.44e+04, v_num=1]
```

Example 1486 (unknown):
```unknown
Epoch 475/500:  95%|█████████▌| 475/500 [54:24<02:51,  6.85s/it, loss=1.46e+04, v_num=1]
```

Example 1487 (unknown):
```unknown
Epoch 476/500:  95%|█████████▌| 475/500 [54:24<02:51,  6.85s/it, loss=1.46e+04, v_num=1]
```

Example 1488 (unknown):
```unknown
Epoch 476/500:  95%|█████████▌| 476/500 [54:31<02:44,  6.87s/it, loss=1.46e+04, v_num=1]
```

Example 1489 (unknown):
```unknown
Epoch 476/500:  95%|█████████▌| 476/500 [54:31<02:44,  6.87s/it, loss=1.45e+04, v_num=1]
```

Example 1490 (unknown):
```unknown
Epoch 477/500:  95%|█████████▌| 476/500 [54:31<02:44,  6.87s/it, loss=1.45e+04, v_num=1]
```

Example 1491 (unknown):
```unknown
Epoch 477/500:  95%|█████████▌| 477/500 [54:38<02:38,  6.90s/it, loss=1.45e+04, v_num=1]
```

Example 1492 (unknown):
```unknown
Epoch 477/500:  95%|█████████▌| 477/500 [54:38<02:38,  6.90s/it, loss=1.46e+04, v_num=1]
```

Example 1493 (unknown):
```unknown
Epoch 478/500:  95%|█████████▌| 477/500 [54:38<02:38,  6.90s/it, loss=1.46e+04, v_num=1]
```

Example 1494 (unknown):
```unknown
Epoch 478/500:  96%|█████████▌| 478/500 [54:44<02:27,  6.72s/it, loss=1.46e+04, v_num=1]
```

Example 1495 (unknown):
```unknown
Epoch 478/500:  96%|█████████▌| 478/500 [54:44<02:27,  6.72s/it, loss=1.43e+04, v_num=1]
```

Example 1496 (unknown):
```unknown
Epoch 479/500:  96%|█████████▌| 478/500 [54:44<02:27,  6.72s/it, loss=1.43e+04, v_num=1]
```

Example 1497 (unknown):
```unknown
Epoch 479/500:  96%|█████████▌| 479/500 [54:50<02:20,  6.67s/it, loss=1.43e+04, v_num=1]
```

Example 1498 (unknown):
```unknown
Epoch 479/500:  96%|█████████▌| 479/500 [54:50<02:20,  6.67s/it, loss=1.45e+04, v_num=1]
```

Example 1499 (unknown):
```unknown
Epoch 480/500:  96%|█████████▌| 479/500 [54:50<02:20,  6.67s/it, loss=1.45e+04, v_num=1]
```

Example 1500 (unknown):
```unknown
Epoch 480/500:  96%|█████████▌| 480/500 [54:56<02:05,  6.26s/it, loss=1.45e+04, v_num=1]
```

Example 1501 (unknown):
```unknown
Epoch 480/500:  96%|█████████▌| 480/500 [54:56<02:05,  6.26s/it, loss=1.45e+04, v_num=1]
```

Example 1502 (unknown):
```unknown
Epoch 481/500:  96%|█████████▌| 480/500 [54:56<02:05,  6.26s/it, loss=1.45e+04, v_num=1]
```

Example 1503 (unknown):
```unknown
Epoch 481/500:  96%|█████████▌| 481/500 [55:02<01:59,  6.28s/it, loss=1.45e+04, v_num=1]
```

Example 1504 (unknown):
```unknown
Epoch 481/500:  96%|█████████▌| 481/500 [55:02<01:59,  6.28s/it, loss=1.45e+04, v_num=1]
```

Example 1505 (unknown):
```unknown
Epoch 482/500:  96%|█████████▌| 481/500 [55:02<01:59,  6.28s/it, loss=1.45e+04, v_num=1]
```

Example 1506 (unknown):
```unknown
Epoch 482/500:  96%|█████████▋| 482/500 [55:06<01:42,  5.71s/it, loss=1.45e+04, v_num=1]
```

Example 1507 (unknown):
```unknown
Epoch 482/500:  96%|█████████▋| 482/500 [55:06<01:42,  5.71s/it, loss=1.47e+04, v_num=1]
```

Example 1508 (unknown):
```unknown
Epoch 483/500:  96%|█████████▋| 482/500 [55:06<01:42,  5.71s/it, loss=1.47e+04, v_num=1]
```

Example 1509 (unknown):
```unknown
Epoch 483/500:  97%|█████████▋| 483/500 [55:11<01:28,  5.22s/it, loss=1.47e+04, v_num=1]
```

Example 1510 (unknown):
```unknown
Epoch 483/500:  97%|█████████▋| 483/500 [55:11<01:28,  5.22s/it, loss=1.43e+04, v_num=1]
```

Example 1511 (unknown):
```unknown
Epoch 484/500:  97%|█████████▋| 483/500 [55:11<01:28,  5.22s/it, loss=1.43e+04, v_num=1]
```

Example 1512 (unknown):
```unknown
Epoch 484/500:  97%|█████████▋| 484/500 [55:15<01:18,  4.89s/it, loss=1.43e+04, v_num=1]
```

Example 1513 (unknown):
```unknown
Epoch 484/500:  97%|█████████▋| 484/500 [55:15<01:18,  4.89s/it, loss=1.44e+04, v_num=1]
```

Example 1514 (unknown):
```unknown
Epoch 485/500:  97%|█████████▋| 484/500 [55:15<01:18,  4.89s/it, loss=1.44e+04, v_num=1]
```

Example 1515 (unknown):
```unknown
Epoch 485/500:  97%|█████████▋| 485/500 [55:19<01:09,  4.65s/it, loss=1.44e+04, v_num=1]
```

Example 1516 (unknown):
```unknown
Epoch 485/500:  97%|█████████▋| 485/500 [55:19<01:09,  4.65s/it, loss=1.45e+04, v_num=1]
```

Example 1517 (unknown):
```unknown
Epoch 486/500:  97%|█████████▋| 485/500 [55:19<01:09,  4.65s/it, loss=1.45e+04, v_num=1]
```

Example 1518 (unknown):
```unknown
Epoch 486/500:  97%|█████████▋| 486/500 [55:23<01:02,  4.48s/it, loss=1.45e+04, v_num=1]
```

Example 1519 (unknown):
```unknown
Epoch 486/500:  97%|█████████▋| 486/500 [55:23<01:02,  4.48s/it, loss=1.47e+04, v_num=1]
```

Example 1520 (unknown):
```unknown
Epoch 487/500:  97%|█████████▋| 486/500 [55:23<01:02,  4.48s/it, loss=1.47e+04, v_num=1]
```

Example 1521 (unknown):
```unknown
Epoch 487/500:  97%|█████████▋| 487/500 [55:27<00:56,  4.36s/it, loss=1.47e+04, v_num=1]
```

Example 1522 (unknown):
```unknown
Epoch 487/500:  97%|█████████▋| 487/500 [55:27<00:56,  4.36s/it, loss=1.44e+04, v_num=1]
```

Example 1523 (unknown):
```unknown
Epoch 488/500:  97%|█████████▋| 487/500 [55:27<00:56,  4.36s/it, loss=1.44e+04, v_num=1]
```

Example 1524 (unknown):
```unknown
Epoch 488/500:  98%|█████████▊| 488/500 [55:31<00:51,  4.28s/it, loss=1.44e+04, v_num=1]
```

Example 1525 (unknown):
```unknown
Epoch 488/500:  98%|█████████▊| 488/500 [55:31<00:51,  4.28s/it, loss=1.47e+04, v_num=1]
```

Example 1526 (unknown):
```unknown
Epoch 489/500:  98%|█████████▊| 488/500 [55:31<00:51,  4.28s/it, loss=1.47e+04, v_num=1]
```

Example 1527 (unknown):
```unknown
Epoch 489/500:  98%|█████████▊| 489/500 [55:35<00:46,  4.22s/it, loss=1.47e+04, v_num=1]
```

Example 1528 (unknown):
```unknown
Epoch 489/500:  98%|█████████▊| 489/500 [55:35<00:46,  4.22s/it, loss=1.44e+04, v_num=1]
```

Example 1529 (unknown):
```unknown
Epoch 490/500:  98%|█████████▊| 489/500 [55:35<00:46,  4.22s/it, loss=1.44e+04, v_num=1]
```

Example 1530 (unknown):
```unknown
Epoch 490/500:  98%|█████████▊| 490/500 [55:39<00:42,  4.21s/it, loss=1.44e+04, v_num=1]
```

Example 1531 (unknown):
```unknown
Epoch 490/500:  98%|█████████▊| 490/500 [55:39<00:42,  4.21s/it, loss=1.45e+04, v_num=1]
```

Example 1532 (unknown):
```unknown
Epoch 491/500:  98%|█████████▊| 490/500 [55:39<00:42,  4.21s/it, loss=1.45e+04, v_num=1]
```

Example 1533 (unknown):
```unknown
Epoch 491/500:  98%|█████████▊| 491/500 [55:43<00:37,  4.17s/it, loss=1.45e+04, v_num=1]
```

Example 1534 (unknown):
```unknown
Epoch 491/500:  98%|█████████▊| 491/500 [55:43<00:37,  4.17s/it, loss=1.45e+04, v_num=1]
```

Example 1535 (unknown):
```unknown
Epoch 491/500:  98%|█████████▊| 491/500 [55:43<01:01,  6.81s/it, loss=1.45e+04, v_num=1]
```

Example 1536 (unknown):
```unknown
Monitored metric reconstruction_loss_validation did not improve in the last 50 records. Best score: 31435.986. Signaling Trainer to stop.
```

Example 1537 (unknown):
```unknown
# save model
if developing==False:
    mvi.save(f'{model_dir}/mvi_{combination}',overwrite=True)
```

Example 1538 (python):
```python
# Extract the latent variables and normalized expression matrices
latent_variables = mvi.get_latent_representation()

# Create new anndata objects with the extracted matrices
latent_variables_adata = anndata.AnnData(X=latent_variables)

# display
print(latent_variables_adata)

# save
if developing==False:
    latent_variables_adata.write(os.path.join(latent_variables_dir, f"mvi_{combination}_latent_variables.h5ad"))
```

Example 1539 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 20
```

Example 1540 (python):
```python
%%time
# add the latent space to adata with original data
adata_mvi.obsm["X_multiVI"] = mvi.get_latent_representation()
```

Example 1541 (unknown):
```unknown
CPU times: user 1.13 s, sys: 1.44 s, total: 2.57 s
Wall time: 2.58 s
```

Example 1542 (python):
```python
%%time
sc.pp.neighbors(adata_mvi,use_rep="X_multiVI")
```

Example 1543 (unknown):
```unknown
computing neighbors
```

Example 1544 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 1545 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 1546 (unknown):
```unknown
finished: added to `.uns['neighbors']`
    `.obsp['distances']`, distances for each pair of neighbors
    `.obsp['connectivities']`, weighted adjacency matrix (0:00:06)
```

Example 1547 (unknown):
```unknown
CPU times: user 5.33 s, sys: 363 ms, total: 5.7 s
Wall time: 6.96 s
```

Example 1548 (python):
```python
%%time
sc.tl.umap(adata_mvi, min_dist=0.1)
```

Example 1549 (unknown):
```unknown
computing UMAP
```

Example 1550 (python):
```python
finished: added
    'X_umap', UMAP coordinates (adata.obsm) (0:00:10)
```

Example 1551 (unknown):
```unknown
CPU times: user 9.89 s, sys: 5.12 ms, total: 9.9 s
Wall time: 10.3 s
```

Example 1552 (python):
```python
if developing==False:
    sc.pl.umap(adata_mvi,title=combination, color=['donor','cell_or_nuclei','kit_10x','scrublet_score','donor', 'week', 'sex', 'region','modality'],
           wspace=0.8,
           ncols=4,
           save=f"_{combination}.png")
else:
    sc.pl.umap(adata_mvi,title=combination, color=['donor','cell_or_nuclei','kit_10x','scrublet_score','donor', 'week', 'sex', 'region','modality'],
           wspace=0.8,
           ncols=4)
```

Example 1553 (unknown):
```unknown
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 1554 (unknown):
```unknown
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 1555 (unknown):
```unknown
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 1556 (unknown):
```unknown
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 1557 (unknown):
```unknown
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 1558 (unknown):
```unknown
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 1559 (unknown):
```unknown
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 1560 (unknown):
```unknown
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 1561 (unknown):
```unknown
WARNING: saving figure to file figures/umap_multivi_aCMs_WholeSample_n_layers1.png
```

Example 1562 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/Visium_mapping/c2l_pt3_plot_immature.html

**Contents:**
- Functions¶
- Read in metadata¶
- Data, parameters¶

2055 rows × 6 columns

2055 rows × 10 columns

2055 rows × 10 columns

**Examples:**

Example 1 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 2 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Global seed set to 0
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 3 (python):
```python
import gc
import torch
```

Example 4 (python):
```python
def read_and_qc(sample_name, path):
    r""" This function reads the data for one 10X spatial experiment into the anndata object.
    It also calculates QC metrics. Modify this function if required by your workflow.

    :param sample_name: Name of the sample
    :param path: path to data
    """

    adata = sc.read_visium(path + str(sample_name),
                           count_file='filtered_feature_bc_matrix.h5', load_images=True)
    adata.var_names_make_unique(join='-')
    
    adata.obs['sample'] = sample_name
    
    # rename genes to ENSEMBL
    adata.var['SYMBOL'] = adata.var_names
    adata.var_names = adata.var['gene_ids']
    adata.var_names.name = None

    # Calculate QC metrics
    sc.pp.calculate_qc_metrics(adata, inplace=True)
    adata.var['mt'] = [gene.startswith('MT-') for gene in adata.var['SYMBOL']]
    adata.obs['mt_frac'] = adata[:, adata.var['mt'].tolist()].X.sum(1).A.squeeze()/adata.obs['total_counts']

    # add sample name to obs names
    adata.obs["sample"] = [str(i) for i in adata.obs['sample']]
    adata.obs_names = adata.obs["sample"] \
                          + '_' + adata.obs_names
    adata.obs.index.name = 'spot_id'

    return adata

def select_slide(adata, s, s_col='sample'):
    r""" This function selects the data for one slide from the spatial anndata object.

    :param adata: Anndata object with multiple spatial experiments
    :param s: name of selected experiment
    :param s_col: column in adata.obs listing experiment name for each location
    """

    slide = adata[adata.obs[s_col].isin([s]), :]
    s_keys = list(slide.uns['spatial'].keys())
    s_spatial = np.array(s_keys)[[s in k for k in s_keys]][0]

    slide.uns['spatial'] = {s_spatial: slide.uns['spatial'][s_spatial]}

    return slide
```

Example 5 (python):
```python
meta=pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230208_for-cell2loc.csv',
                index_col=0)
meta
```

Example 6 (python):
```python
pd.crosstab(meta['donor'],meta['cell2loc-map_group'])
```

Example 7 (unknown):
```unknown
# select slides
# Healthy, OCT, and without DA
meta = meta[meta['cell2loc-map_group'].isin(['4PCW','5PCW','7PCW','13PCW','20PCW'])]

sampleIDs = list(meta['sangerID'])
len(sampleIDs)
```

Example 8 (unknown):
```unknown
run = "20Mar2024_fetal_OCT"
```

Example 9 (python):
```python
import os
results_folder = os.path.join(os.getcwd(),"analysis/Visium/cell2location_results")

ref_run_name = f'{results_folder}/reference_signatures/{run}'
run_name = f'{results_folder}/cell2location_map/{run}'
```

Example 10 (unknown):
```unknown
visium_path='/nfs/team205/heart/mapped/spaceranger110/'

celltype_key='fine_grain_modified'

vis_subset_key='cell2loc-map_group'
```

Example 11 (python):
```python
%%time
subset_values = os.listdir(os.path.join(results_folder, "cell2location_map", run))
print(f"Found these subsets {subset_values}")

for value in subset_values:
    print(f"###### {value} ######")
    adata_sub = sc.read(f"{run_name}/{value}/sp.h5ad")
    print("visium read in")
    adata_sub.obs[adata_sub.uns['mod']['factor_names']] = adata_sub.obsm['q05_cell_abundance_w_sf']
    IDs = meta[meta[vis_subset_key] == value].sangerID.tolist()
    print(f"{value}: {IDs}")

    for sample in IDs:
        try:
            slide = select_slide(adata_sub, sample)
        except IndexError as e:
            print(f"Sample {sample} not found in adata: {e}")
            continue  # Skip this sample if not found

        slide.obs[slide.uns['mod']['factor_names']] = slide.obsm['q05_cell_abundance_w_sf']
        list_of_cell_types = slide.uns['mod']['factor_names'].tolist()
        print("plotting HE image...")
        sc.pl.spatial(slide, color=None, size=0, img_key='hires', title=f"HE:{sample}", 
                      save=f"{run}__{value}__{sample}__HE.png", show=False)

        # Plotting abundances for all relevant cell states
        if list_of_cell_types:
            print("plotting abundances for all cell states...")
            sc.pl.spatial(slide, color=list_of_cell_types, cmap="RdPu", ncols=10, size=1.3, 
                          img_key='hires', vmin=0, vmax='p99.2', alpha_img=0.3, 
                          save=f"{run}__{value}__{sample}__all_cell_states_abundances.png", show=False)
```

Example 12 (unknown):
```unknown
Found these subsets ['5PCW', '20PCW', '13PCW', '4PCW', '7PCW']
###### 5PCW ######
visium read in
5PCW: ['HCAHeartST11605167', 'HCAHeartST11605168']
```

Example 13 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 14 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__5PCW__HCAHeartST11605167__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__5PCW__HCAHeartST11605167__all_cell_states_abundances.png
```

Example 15 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 16 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__5PCW__HCAHeartST11605168__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__5PCW__HCAHeartST11605168__all_cell_states_abundances.png
###### 20PCW ######
visium read in
20PCW: ['HCAHeartST13162329', 'HCAHeartST13162330', 'HCAHeartST13162331']
```

Example 17 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 18 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__20PCW__HCAHeartST13162329__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__20PCW__HCAHeartST13162329__all_cell_states_abundances.png
```

Example 19 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 20 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__20PCW__HCAHeartST13162330__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__20PCW__HCAHeartST13162330__all_cell_states_abundances.png
```

Example 21 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 22 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__20PCW__HCAHeartST13162331__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__20PCW__HCAHeartST13162331__all_cell_states_abundances.png
###### 13PCW ######
visium read in
13PCW: ['HCAHeartST11605165', 'HCAHeartST11605166']
```

Example 23 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 24 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__13PCW__HCAHeartST11605165__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__13PCW__HCAHeartST11605165__all_cell_states_abundances.png
```

Example 25 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 26 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__13PCW__HCAHeartST11605166__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__13PCW__HCAHeartST11605166__all_cell_states_abundances.png
###### 4PCW ######
visium read in
4PCW: ['HCAHeartST13162332']
```

Example 27 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 28 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__4PCW__HCAHeartST13162332__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__4PCW__HCAHeartST13162332__all_cell_states_abundances.png
###### 7PCW ######
visium read in
7PCW: ['HCAHeartST10298114', 'HCAHeartST10298115', 'HCAHeartST10298116', 'HCAHeartST10298117', 'HCAHeartST10298120', 'HCAHeartST10317184', 'HCAHeartST10317185', 'HCAHeartST10317186', 'HCAHeartST10317189', 'HCAHeartST10317190']
```

Example 29 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 30 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10298114__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10298114__all_cell_states_abundances.png
```

Example 31 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 32 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10298115__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10298115__all_cell_states_abundances.png
```

Example 33 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 34 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10298116__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10298116__all_cell_states_abundances.png
```

Example 35 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 36 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10298117__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10298117__all_cell_states_abundances.png
```

Example 37 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 38 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10298120__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10298120__all_cell_states_abundances.png
```

Example 39 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 40 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10317184__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10317184__all_cell_states_abundances.png
```

Example 41 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 42 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10317185__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10317185__all_cell_states_abundances.png
```

Example 43 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 44 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10317186__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10317186__all_cell_states_abundances.png
```

Example 45 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 46 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10317189__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10317189__all_cell_states_abundances.png
```

Example 47 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 48 (unknown):
```unknown
plotting HE image...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10317190__HE.png
plotting abundances for all cell states...
WARNING: saving figure to file figures/show20Mar2024_fetal_OCT__7PCW__HCAHeartST10317190__all_cell_states_abundances.png
CPU times: user 9min 45s, sys: 2min 46s, total: 12min 31s
Wall time: 12min 40s
```

Example 49 (python):
```python
%%time

adata_dict={}

subset_values = os.listdir(os.path.join(results_folder, "cell2location_map", run))
print(f"Found these subsets {subset_values}")

for value in subset_values:
    print(f"###### {value} ######")
    adata_sub = sc.read(f"{run_name}/{value}/sp.h5ad")
    print("visium read in")
    adata_sub.obs[adata_sub.uns['mod']['factor_names']] = adata_sub.obsm['q05_cell_abundance_w_sf']
    IDs = meta[meta[vis_subset_key] == value].sangerID.tolist()
    print(f"{value}: {IDs}")

    for sample in IDs:
        try:
            slide = select_slide(adata_sub, sample)
        except IndexError as e:
            print(f"Sample {sample} not found in adata: {e}")
            continue  # Skip this sample if not found

        slide.obs[slide.uns['mod']['factor_names']] = slide.obsm['q05_cell_abundance_w_sf']
        adata_dict[sample]=slide
```

Example 50 (unknown):
```unknown
Found these subsets ['5PCW', '20PCW', '13PCW', '4PCW', '7PCW']
###### 5PCW ######
visium read in
5PCW: ['HCAHeartST11605167', 'HCAHeartST11605168']
```

Example 51 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 52 (unknown):
```unknown
###### 20PCW ######
visium read in
20PCW: ['HCAHeartST13162329', 'HCAHeartST13162330', 'HCAHeartST13162331']
```

Example 53 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 54 (unknown):
```unknown
###### 13PCW ######
visium read in
13PCW: ['HCAHeartST11605165', 'HCAHeartST11605166']
```

Example 55 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 56 (unknown):
```unknown
###### 4PCW ######
visium read in
4PCW: ['HCAHeartST13162332']
```

Example 57 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 58 (unknown):
```unknown
###### 7PCW ######
visium read in
7PCW: ['HCAHeartST10298114', 'HCAHeartST10298115', 'HCAHeartST10298116', 'HCAHeartST10298117', 'HCAHeartST10298120', 'HCAHeartST10317184', 'HCAHeartST10317185', 'HCAHeartST10317186', 'HCAHeartST10317189', 'HCAHeartST10317190']
```

Example 59 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
```

Example 60 (unknown):
```unknown
CPU times: user 3.58 s, sys: 8.52 s, total: 12.1 s
Wall time: 18.3 s
```

Example 61 (python):
```python
adata_dict.keys()
```

Example 62 (unknown):
```unknown
dict_keys(['HCAHeartST11605167', 'HCAHeartST11605168', 'HCAHeartST13162329', 'HCAHeartST13162330', 'HCAHeartST13162331', 'HCAHeartST11605165', 'HCAHeartST11605166', 'HCAHeartST13162332', 'HCAHeartST10298114', 'HCAHeartST10298115', 'HCAHeartST10298116', 'HCAHeartST10298117', 'HCAHeartST10298120', 'HCAHeartST10317184', 'HCAHeartST10317185', 'HCAHeartST10317186', 'HCAHeartST10317189', 'HCAHeartST10317190'])
```

Example 63 (python):
```python
from tqdm import tqdm
atrial_data_dict={}
for sample,tmp_adata in tqdm(adata_dict.items()):
    print(f"sample: {sample},  age is {tmp_adata.obs.age.unique().tolist()[0]}")
    tmp_df=tmp_adata[tmp_adata.obs.JC_edit=="Myocardium - Atrial"].obs[['AtrialCardiomyocytesLeft_immature', 'AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight','AtrialCardiomyocytesRight_immature']]

    if tmp_df.shape[0]>0:
        tmp_df["sample"]=sample
        tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
        atrial_data_dict[sample]=tmp_df
    else:
        print(f"no atrial myocardium spots annotated for sample: {sample}")
```

Example 64 (python):
```python
0%|                                                                                                                                                                                                                    | 0/18 [00:00<?, ?it/s]/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
 67%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                                                   | 12/18 [00:00<00:00, 109.51it/s]/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
/tmp/ipykernel_167807/3397130741.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["sample"]=sample
/tmp/ipykernel_167807/3397130741.py:9: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  tmp_df["age"]=tmp_adata.obs.age.unique().tolist()[0]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:00<00:00, 109.20it/s]
```

Example 65 (unknown):
```unknown
sample: HCAHeartST11605167,  age is GA-7w4d
sample: HCAHeartST11605168,  age is GA-7w4d
sample: HCAHeartST13162329,  age is PCW-20w
no atrial myocardium spots annotated for sample: HCAHeartST13162329
sample: HCAHeartST13162330,  age is PCW-20w
no atrial myocardium spots annotated for sample: HCAHeartST13162330
sample: HCAHeartST13162331,  age is PCW-20w
no atrial myocardium spots annotated for sample: HCAHeartST13162331
sample: HCAHeartST11605165,  age is PCW-13w
sample: HCAHeartST11605166,  age is PCW-13w
sample: HCAHeartST13162332,  age is GA-6w3d
no atrial myocardium spots annotated for sample: HCAHeartST13162332
sample: HCAHeartST10298114,  age is GA-9w4d
sample: HCAHeartST10298115,  age is GA-9w4d
sample: HCAHeartST10298116,  age is GA-9w4d
sample: HCAHeartST10298117,  age is GA-9w4d
sample: HCAHeartST10298120,  age is GA-9w4d
sample: HCAHeartST10317184,  age is GA-9w4d
sample: HCAHeartST10317185,  age is GA-9w4d
sample: HCAHeartST10317186,  age is GA-9w4d
sample: HCAHeartST10317189,  age is GA-9w4d
sample: HCAHeartST10317190,  age is GA-9w4d
```

Example 66 (python):
```python
data=pd.concat(list(atrial_data_dict.values()))
data
```

Example 67 (unknown):
```unknown
data.to_csv("AtrialProportionsOCT.csv")
```

Example 68 (unknown):
```unknown
# Assuming 'data' is your pandas DataFrame
# Calculate the total for each row
total = data['AtrialCardiomyocytesLeft_immature'] + data['AtrialCardiomyocytesLeft'] + data['AtrialCardiomyocytesRight'] + data['AtrialCardiomyocytesRight_immature']

# Calculate the proportions for each column
data['AtrialCardiomyocytesRight_immature_prop'] = data['AtrialCardiomyocytesRight_immature'] / total
data['AtrialCardiomyocytesLeft_immature_prop'] = data['AtrialCardiomyocytesLeft_immature'] / total
data['AtrialCardiomyocytesRight_prop'] = data['AtrialCardiomyocytesRight'] / total
data['AtrialCardiomyocytesLeft_prop'] = data['AtrialCardiomyocytesLeft'] / total

# The new DataFrame 'data' now includes the proportion columns
data
```

Example 69 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt

# Assuming 'data' is your DataFrame and it already includes an 'age' column
df_long = data.melt(id_vars='age', var_name='Cell Type', value_name='Proportion', 
                    value_vars=['AtrialCardiomyocytesLeft_immature_prop', 'AtrialCardiomyocytesLeft_prop', 
                                'AtrialCardiomyocytesRight_prop', 'AtrialCardiomyocytesRight_immature_prop'])

# Create the box plot with 'hue' for age categories
plt.figure(figsize=(12, 8))
sns.boxplot(x='age', y='Proportion', hue='Cell Type', data=df_long)
plt.title('Distribution of Cell Type Proportions Across Age Categories')
plt.xticks(rotation=45)
plt.xlabel('Cell Type')
plt.ylabel('Proportion')
plt.legend(title='Age Category', bbox_to_anchor=(1.05, 1), loc='upper left')
plt.tight_layout()
plt.show()
```

Example 70 (python):
```python
# Assuming 'data' is your pandas DataFrame
# Calculate the total for Left and Right cells separately
total_left = data['AtrialCardiomyocytesLeft_immature'] + data['AtrialCardiomyocytesLeft']
total_right = data['AtrialCardiomyocytesRight_immature'] + data['AtrialCardiomyocytesRight']

# Calculate the proportions for each cell type relative to their side
data['AtrialCardiomyocytesLeft_immature_prop'] = data['AtrialCardiomyocytesLeft_immature'] / total_left
data['AtrialCardiomyocytesLeft_prop'] = data['AtrialCardiomyocytesLeft'] / total_left

data['AtrialCardiomyocytesRight_immature_prop'] = data['AtrialCardiomyocytesRight_immature'] / total_right
data['AtrialCardiomyocytesRight_prop'] = data['AtrialCardiomyocytesRight'] / total_right

# Check the new columns in the DataFrame
data[['AtrialCardiomyocytesLeft_immature_prop', 'AtrialCardiomyocytesLeft_prop', 'AtrialCardiomyocytesRight_immature_prop', 'AtrialCardiomyocytesRight_prop']]

import numpy as np

data['AtrialCardiomyocytesLeft_immature_prop'] = np.where(total_left == 0, 0, data['AtrialCardiomyocytesLeft_immature'] / total_left)
data['AtrialCardiomyocytesLeft_prop'] = np.where(total_left == 0, 0, data['AtrialCardiomyocytesLeft'] / total_left)

data['AtrialCardiomyocytesRight_immature_prop'] = np.where(total_right == 0, 0, data['AtrialCardiomyocytesRight_immature'] / total_right)
data['AtrialCardiomyocytesRight_prop'] = np.where(total_right == 0, 0, data['AtrialCardiomyocytesRight'] / total_right)
data
```

Example 71 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt

# Assuming 'data' is your DataFrame and it already includes an 'age' column
df_long = data.melt(id_vars='age', var_name='Cell Type', value_name='Proportion', 
                    value_vars=['AtrialCardiomyocytesLeft_immature_prop', 'AtrialCardiomyocytesLeft_prop', 
                                'AtrialCardiomyocytesRight_prop', 'AtrialCardiomyocytesRight_immature_prop'])

# Create the box plot with 'hue' for age categories
plt.figure(figsize=(12, 8))
sns.boxplot(x='age', y='Proportion', hue='Cell Type', data=df_long)
plt.title('Distribution of Cell Type Proportions Across Age Categories')
plt.xticks(rotation=45)
plt.xlabel('Cell Type')
plt.ylabel('Proportion')
plt.legend(title='Age Category', bbox_to_anchor=(1.05, 1), loc='upper left')
plt.tight_layout()
plt.show()
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_ventricularCM.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Markers¶
- Subset "uncertain" cells¶
- Subset uncertain cells - 2nd round¶
- Add the annotation to the original adata¶
- Save¶

2366 rows × 7 columns

--> cluster 1 has low total_counts and bin_counts. and no specific marker expression --> will annotated as "unassigned"

--> cluster 1 looks like VCS-distal

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'VentricularCardiomyocytes'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 61075 × 17994
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers
```

Example 10 (unknown):
```unknown
'GATA2' in list(midgrain_markers['names'])
```

Example 11 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 12 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 61075 × 16746
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 13 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 14 (unknown):
```unknown
number of hvgs: 2241
```

Example 15 (python):
```python
'HEY2' in adata.var_names
```

Example 16 (python):
```python
markers = {
    'VCSprox':['TBX3','PRDM16','NIBAN1','PHACTR1','FGF18'], # 'PRDM16','NIBAN1','PHACTR1','FGF18' are the markers compared to VCSdistal
    'VCSdist':['SEMA3A','IRX2','IRX3','OPCML','NTN1','EPHA4','GABRB2'],
    'Trab':['ADAMTSL1','SGCD','CSMD1'],
    'Compact':['TBX20','HEY2'],
    'Cycling':['MKI67','TOP2A']
}

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in adata.var_names)&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata.var_names]
len(genes)
```

Example 17 (python):
```python
sc.pl.embedding(adata,basis='X_umap',color=genes, size=10,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 18 (unknown):
```unknown
resolution_sel = 1
```

Example 19 (python):
```python
markers_others = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers_others.keys():
    markers_others[key]=[x for x in markers_others[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers_others, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 20 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 21 (python):
```python
sc.tl.dendrogram(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby=f'leiden_{str(resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 22 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: VCSprox, VCSdist, Trab, etc.
```

Example 23 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 24 (unknown):
```unknown
[1.47180343 1.33566585 1.26948215 1.24032132 1.26104042]
```

Example 25 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 26 (unknown):
```unknown
[1.47180343 1.33566585 1.26948215 1.24032132 1.26104042]
```

Example 27 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 28 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 29 (python):
```python
# umap per celltype to highlight
group_col = 'celltypist_midmod2fine'
groups_to_plot = list(set(adata.obs[group_col]))

plt.rcParams['axes.grid'] = False
fig,axs=plt.subplots(nrows=2,ncols=4,figsize=(15,7))
# Flatten the axs array for easier iteration in case of multiple rows and columns
axs_flat = axs.flatten()

for i,grp in enumerate(groups_to_plot):
    sc.pl.umap(adata, color=[group_col], groups=grp, 
               size=10, frameon=False,
               show=False, ax=axs_flat[i], legend_loc=None, legend_fontsize=5,
               title=grp
              )
# Turn off any remaining subplots
for j in range(len(groups_to_plot), len(axs_flat)):
    fig.delaxes(axs_flat[j])

# fig.suptitle('Main Title for the Figure')
fig.tight_layout()
```

Example 30 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 31 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 32 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'VentricularCardiomyocytesCompact',
    '1':'unassigned',
    '2':'uncertain',
    '3':'uncertain',
    '4':'uncertain',
    '5':'VentricularConductionSystemProximal',
    '6':'VentricularConductionSystemProximal',
    '7':'VentricularCardiomyocytes_DoubletsWithErythrocytes',
    '8':'VentricularCardiomyocytes_Doublets',
    '9':'uncertain',
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 33 (unknown):
```unknown
fine_grain_manual
uncertain                                             27579
VentricularCardiomyocytesCompact                      13082
unassigned                                            12767
VentricularConductionSystemProximal                    5902
VentricularCardiomyocytes_DoubletsWithErythrocytes     1723
VentricularCardiomyocytes_Doublets                       22
Name: count, dtype: int64
```

Example 34 (unknown):
```unknown
'0':'VentricularCardiomyocytesCompact',
    '1':'VentricularCardiomyocytesCompact',
    '2':'VentricularCardiomyocytesTrabeculated',
    '3':'VentricularConductionSystem',
```

Example 35 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
set(ad.obs['fine_grain'])
```

Example 36 (unknown):
```unknown
{'AtrialCardiomyocytesCycling',
 'AtrialCardiomyocytesLeft',
 'AtrialCardiomyocytesRight',
 'AtrioventricularNodePacemakerCells',
 'BCells',
 'BCellsMS4A1pos',
 'ChromaffinCells',
 'CoronaryArterialEndothelialCells',
 'CoronaryCapillaryEndothelialCells',
 'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'CoronaryVenousEndothelialCells',
 'CoronaryVesselAdventitialFibroblasts',
 'DendriticCellsMature',
 'DendriticCellsType1',
 'DuctusArteriosusSmoothMuscleCells',
 'EndocardialCells',
 'EndocardialCushionCells',
 'EpicardiumDerivedCells',
 'GreatVesselAdventitialFibroblasts',
 'GreatVesselArterialEndothelialCells',
 'GreatVesselSmoothMuscleCells',
 'GreatVesselVenousEndothelialCells',
 'InnateLymphoidCells',
 'LymphNodeFibroblasticReticularCells',
 'LymphaticEndothelialCells',
 'MacrophagesATF3pos',
 'MacrophagesCX3CR1pos',
 'MacrophagesLYVE1pos',
 'MacrophagesTIMD4pos',
 'MastCells',
 'Megakaryocytes',
 'MesothelialEpicardialCells',
 'MonocyteDerivedCells',
 'Monocytes',
 'MonocytesMPOpos',
 'MyocardialInterstitialFibroblasts',
 'Myofibroblasts',
 'NaturalKillerCells',
 'NeuronPrecursors',
 'ParasympatheticNeurons',
 'PericardialCellsFibrous',
 'PericardialCellsIntermediate',
 'PericardialCellsParietal',
 'PlasmacytoidDendriticCells',
 'ProBCells',
 'SchwannCellPrecursors',
 'SchwannCells',
 'SinoatrialNodePacemakerCells',
 'SubEpicardialFibroblasts',
 'SympatheticNeurons',
 'TCellsCD4pos',
 'TCellsCD8pos',
 'TregsCD4pos',
 'ValveEndothelialCells',
 'ValveInterstitialCells',
 'VentricularCardiomyocytesCycling',
 'VentricularCardiomyocytesLeftCompact',
 'VentricularCardiomyocytesLeftTrabeculated',
 'VentricularCardiomyocytesRightCompact',
 'VentricularCardiomyocytesRightTrabeculated',
 'VentricularConductionSystemDistal',
 'VentricularConductionSystemProximal'}
```

Example 37 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 38 (python):
```python
sc.tl.dendrogram(adata,groupby='fine_grain_manual')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 39 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: VentricularCardiomyocytesCompact, unassigned, uncertain, etc.
var_group_labels: VCSprox, VCSdist, Trab, etc.
```

Example 40 (python):
```python
set(adata.obs['fine_grain_manual'])
```

Example 41 (unknown):
```unknown
{'VentricularCardiomyocytesCompact',
 'VentricularCardiomyocytes_Doublets',
 'VentricularCardiomyocytes_DoubletsWithErythrocytes',
 'VentricularConductionSystemProximal',
 'unassigned',
 'uncertain'}
```

Example 42 (python):
```python
adata_sub = adata[adata.obs['fine_grain_manual'].isin(['uncertain'])]

# drop leiden clustering results
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    adata_sub.obs.drop(f'leiden_{str(resolution)}',axis=1,inplace=True)
adata_sub
```

Example 43 (unknown):
```unknown
AnnData object with n_obs × n_vars = 27579 × 17994
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap', 'dendrogram_leiden_1', 'rank_genes_groups', 'fine_grain_manual_colors', 'dendrogram_fine_grain_manual'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 44 (python):
```python
# remove mid_grain markers
mask = [x not in list(midgrain_markers['names']) for x in adata_sub.var_names]
bdata = adata_sub[:,mask]
bdata
```

Example 45 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 27579 × 16746
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap', 'dendrogram_leiden_1', 'rank_genes_groups', 'fine_grain_manual_colors', 'dendrogram_fine_grain_manual'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 46 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata_sub.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata_sub,n_neighbors=15,use_rep='X_pca')
```

Example 47 (unknown):
```unknown
number of hvgs: 2182
```

Example 48 (python):
```python
sc.tl.umap(adata_sub,min_dist=0.1)
```

Example 49 (python):
```python
%%time
# reclustering
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata_sub,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata_sub,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 50 (unknown):
```unknown
CPU times: user 17.1 s, sys: 466 ms, total: 17.5 s
Wall time: 18.1 s
```

Example 51 (python):
```python
markers = {
    # 'VCSprox':['TBX3','PRDM16','NIBAN1','PHACTR1','FGF18'], # 'PRDM16','NIBAN1','PHACTR1','FGF18' are the markers compared to VCSdistal
    'VCSdist':['SEMA3A','IRX2','IRX3','OPCML','NTN1','EPHA4','GABRB2'],
    'Trab':['ADAMTSL1','SGCD','CSMD1'],
    'Compact':['TBX20','HEY2'],
    'Cycling':['MKI67','TOP2A']
}

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in adata.var_names)&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata.var_names]
len(genes)
```

Example 52 (python):
```python
sc.pl.embedding(adata_sub,basis='X_umap',color=genes, size=10,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 53 (python):
```python
adata_sub_resolution_sel = 0.6
```

Example 54 (python):
```python
markers_others = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers_others.keys():
    markers_others[key]=[x for x in markers_others[key] if x in adata_sub.var_names]

sc.tl.dendrogram(adata_sub, groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub,
              markers_others, 
              groupby=f'leiden_{str(adata_sub_resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 55 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2
var_group_labels: CM, FB, EC, etc.
```

Example 56 (python):
```python
sc.tl.dendrogram(adata_sub,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub,
              var_names=markers,
              groupby=f'leiden_{str(adata_sub_resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 57 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2
var_group_labels: VCSdist, Trab, Compact, etc.
```

Example 58 (python):
```python
# DEG analysis
print(adata_sub.X.data[:5])
adata_sub.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata_sub,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.tl.dendrogram(adata_sub, groupby=f'leiden_{str(adata_sub_resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata_sub, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 59 (unknown):
```unknown
[2.16681224 1.99582797 1.93329773 1.91105105 1.8337824 ]
```

Example 60 (python):
```python
# DEG analysis
print(adata_sub.X.data[:5])
adata_sub.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata_sub,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.tl.dendrogram(adata_sub, groupby=f'leiden_{str(adata_sub_resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata_sub, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 61 (unknown):
```unknown
[2.16681224 1.99582797 1.93329773 1.91105105 1.8337824 ]
```

Example 62 (python):
```python
sc.pl.embedding(adata_sub,basis='X_umap',
                color=f'leiden_{str(adata_sub_resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 63 (python):
```python
# annotate
adata_sub.obs['fine_grain_manual'] = adata_sub.obs[f'leiden_{str(adata_sub_resolution_sel)}'].copy()
adata_sub.obs.replace({'fine_grain_manual':{
    '0':'VentricularCardiomyocytesCompact', # cluster 0 has more FB and vasculature signatures compared to cluster 1. Potentially there's some level of contamination. Which also suggests these population are more close to FB/Vasc.
    '1':'VentricularCardiomyocytesCompact',
    '2':'VentricularConductionSystemDistal_VentricularCardiomyocytesTrabeculated',
}},inplace=True)
adata_sub.obs['fine_grain_manual'].value_counts()
```

Example 64 (unknown):
```unknown
fine_grain_manual
VentricularCardiomyocytesCompact                                           19888
VentricularConductionSystemDistal_VentricularCardiomyocytesTrabeculated     7691
Name: count, dtype: int64
```

Example 65 (python):
```python
sc.pl.embedding(adata_sub,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 66 (python):
```python
sc.tl.dendrogram(adata_sub,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 67 (unknown):
```unknown
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 68 (python):
```python
adata_sub2 = adata_sub[adata_sub.obs['fine_grain_manual'].isin(['VentricularConductionSystemDistal_VentricularCardiomyocytesTrabeculated'])]

# drop leiden clustering results
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    adata_sub2.obs.drop(f'leiden_{str(resolution)}',axis=1,inplace=True)
adata_sub2
```

Example 69 (unknown):
```unknown
AnnData object with n_obs × n_vars = 7691 × 17994
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap', 'dendrogram_leiden_1', 'rank_genes_groups', 'fine_grain_manual_colors', 'dendrogram_fine_grain_manual', 'leiden_0.2_colors', 'leiden_0.4_colors', 'leiden_0.6_colors', 'leiden_1.2_colors', 'dendrogram_leiden_0.6'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 70 (python):
```python
# remove mid_grain markers
mask = [x not in list(midgrain_markers['names']) for x in adata_sub2.var_names]
bdata = adata_sub2[:,mask]
bdata
```

Example 71 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 7691 × 16746
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'hvg', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'spatial', 'umap', 'dendrogram_leiden_1', 'rank_genes_groups', 'fine_grain_manual_colors', 'dendrogram_fine_grain_manual', 'leiden_0.2_colors', 'leiden_0.4_colors', 'leiden_0.6_colors', 'leiden_1.2_colors', 'dendrogram_leiden_0.6'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 72 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata_sub2.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata_sub2,n_neighbors=15,use_rep='X_pca')
```

Example 73 (unknown):
```unknown
number of hvgs: 3227
```

Example 74 (python):
```python
sc.tl.umap(adata_sub2,min_dist=0.1)
```

Example 75 (python):
```python
%%time
# reclustering
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata_sub2,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata_sub2,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 76 (unknown):
```unknown
CPU times: user 6.04 s, sys: 148 ms, total: 6.19 s
Wall time: 6.37 s
```

Example 77 (python):
```python
markers = {
    # 'VCSprox':['TBX3','PRDM16','NIBAN1','PHACTR1','FGF18'], # 'PRDM16','NIBAN1','PHACTR1','FGF18' are the markers compared to VCSdistal
    'VCSdist':['SEMA3A','IRX2','IRX3','OPCML','NTN1','EPHA4','GABRB2'],
    # 'Trab':['ADAMTSL1','SGCD','CSMD1'],
    # 'Compact':['TBX20','HEY2'],
    'Cycling':['MKI67','TOP2A']
}

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in adata_sub2.var_names)&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata_sub2.var_names]
len(genes)
```

Example 78 (python):
```python
sc.pl.embedding(adata_sub2,basis='X_umap',color=genes, size=10,
               ncols=6, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 79 (python):
```python
adata_sub_resolution_sel = 1
```

Example 80 (python):
```python
markers_others = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers_others.keys():
    markers_others[key]=[x for x in markers_others[key] if x in adata_sub2.var_names]

sc.tl.dendrogram(adata_sub2, groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub2,
              markers_others, 
              groupby=f'leiden_{str(adata_sub_resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 81 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 82 (python):
```python
sc.tl.dendrogram(adata_sub2,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub2,
              var_names=markers,
              groupby=f'leiden_{str(adata_sub_resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 83 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: VCSdist, Cycling
```

Example 84 (python):
```python
# DEG analysis
print(adata_sub2.X.data[:5])
adata_sub2.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata_sub2,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.tl.dendrogram(adata_sub2, groupby=f'leiden_{str(adata_sub_resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata_sub2, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 85 (unknown):
```unknown
[2.71216227 1.03967184 1.35760811 1.70411582 1.28981545]
```

Example 86 (python):
```python
# DEG analysis
print(adata_sub2.X.data[:5])
adata_sub2.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata_sub2,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.tl.dendrogram(adata_sub2, groupby=f'leiden_{str(adata_sub_resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata_sub2, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 87 (unknown):
```unknown
[2.71216227 1.03967184 1.35760811 1.70411582 1.28981545]
```

Example 88 (python):
```python
sc.pl.embedding(adata_sub2,basis='X_umap',
                color=f'leiden_{str(adata_sub_resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 89 (python):
```python
# annotate
adata_sub2.obs['fine_grain_manual'] = adata_sub2.obs[f'leiden_{str(adata_sub_resolution_sel)}'].copy()
adata_sub2.obs.replace({'fine_grain_manual':{
    '0':'VentricularCardiomyocytesTrabeculated',
    '1':'VentricularConductionSystemDistal',
    '2':'VentricularCardiomyocytesTrabeculated',
    '3':'VentricularCardiomyocytesTrabeculated',
    '4':'VentricularCardiomyocytesTrabeculated',
    '5':'VentricularCardiomyocytesTrabeculated',
    '6':'VentricularCardiomyocytesTrabeculated',
    '7':'VentricularCardiomyocytesTrabeculated',
    '8':'VentricularCardiomyocytesTrabeculated',
    '9':'VentricularCardiomyocytesTrabeculated',
}},inplace=True)
adata_sub2.obs['fine_grain_manual'].value_counts()
```

Example 90 (unknown):
```unknown
fine_grain_manual
VentricularCardiomyocytesTrabeculated    6349
VentricularConductionSystemDistal        1342
Name: count, dtype: int64
```

Example 91 (python):
```python
sc.pl.embedding(adata_sub2,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 92 (python):
```python
sc.tl.dendrogram(adata_sub2,groupby=f'leiden_{str(adata_sub_resolution_sel)}')
sc.pl.dotplot(adata_sub2,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 93 (unknown):
```unknown
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 94 (python):
```python
# add the annotation to the original adata
adata.obs['fine_grain_manual'] = adata.obs['fine_grain_manual'].astype('str')
adata.obs.loc[adata_sub.obs_names,'fine_grain_manual'] = adata_sub.obs['fine_grain_manual'].astype('str').copy()
adata.obs.loc[adata_sub2.obs_names,'fine_grain_manual'] = adata_sub2.obs['fine_grain_manual'].astype('str').copy()
adata.obs['fine_grain_manual'].value_counts()
```

Example 95 (unknown):
```unknown
fine_grain_manual
VentricularCardiomyocytesCompact                      32970
unassigned                                            12767
VentricularCardiomyocytesTrabeculated                  6349
VentricularConductionSystemProximal                    5902
VentricularCardiomyocytes_DoubletsWithErythrocytes     1723
VentricularConductionSystemDistal                      1342
VentricularCardiomyocytes_Doublets                       22
Name: count, dtype: int64
```

Example 96 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 97 (unknown):
```unknown
... storing 'fine_grain_manual' as categorical
```

Example 98 (python):
```python
markers = {
    'VCSprox':['TBX3','PRDM16','NIBAN1','PHACTR1','FGF18'], # 'PRDM16','NIBAN1','PHACTR1','FGF18' are the markers compared to VCSdistal
    'VCSdist':['SEMA3A','IRX2','IRX3','OPCML','NTN1','EPHA4','GABRB2'],
    'Trab':['ADAMTSL1','SGCD','CSMD1'],
    'Compact':['TBX20','HEY2'],
    'Cycling':['MKI67','TOP2A']
}

genes = []
for g in markers.values():
    genes = genes + [x for x in g if (x in adata_sub2.var_names)&(x not in genes)]
for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata_sub2.var_names]
len(genes)
```

Example 99 (python):
```python
sc.tl.dendrogram(adata,groupby='fine_grain_manual')
sc.pl.dotplot(adata,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 100 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: VentricularCardiomyocytesCompact, VentricularCardiomyocytesTrabeculated, VentricularCardiomyocytes_Doublets, etc.
var_group_labels: VCSprox, VCSdist, Trab, etc.
```

Example 101 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

Example 102 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets/HEA_FOET14880396_VentricularCardiomyocytes_b2c_cells_filtered_lognorm.h5ad
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no7_manual-annotation_atrialCM.html

**Contents:**
- Variables¶
- Read in data¶
- Remove mid-grain markers from the feature¶
- PCA, Clustering, UMAP¶
- Subset SAN cells¶
- Save¶
- Read in for plotting¶

2378 rows × 7 columns

--> cluster 1 has low total_counts and bin_counts. and no specific marker expression --> will annotated as "unassigned"

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 

import celltypist
```

Example 2 (python):
```python
# import session_info
# session_info.show()
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=80)
```

Example 5 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets'
sample_id = 'HEA_FOET14880396'
celltype = 'AtrialCardiomyocytes'
```

Example 6 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 7 (python):
```python
# read in
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 13904 × 17650
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'dendrogram_fine_grain_manual', 'dendrogram_leiden_1', 'dendrogram_leiden_1.5', 'fine_grain_manual_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'rank_genes_groups', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
# read in
midgrain_markers = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/VisiumHD/re-seq_mpp0p3/sc_midgrain_markers.csv',
                               index_col=0
                              )
# remove the markers of celltype of interest
midgrain_markers = midgrain_markers[midgrain_markers['group']!=celltype]
midgrain_markers
```

Example 10 (unknown):
```unknown
'GATA2' in list(midgrain_markers['names'])
```

Example 11 (python):
```python
# remove
mask = [x not in list(midgrain_markers['names']) for x in adata.var_names]
bdata = adata[:,mask]
bdata
```

Example 12 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 13904 × 16391
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'dendrogram_fine_grain_manual', 'dendrogram_leiden_1', 'dendrogram_leiden_1.5', 'fine_grain_manual_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2', 'log1p', 'neighbors', 'pca', 'rank_genes_groups', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 13 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
adata.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(adata,n_neighbors=15,use_rep='X_pca')
# visualisation with umap
sc.tl.umap(adata,min_dist=0.5)
sc.pl.embedding(adata,basis='X_umap',color=['bin_count','total_counts','n_genes_by_counts',"conf_score_midmod2fine","celltypist_midmod2fine",],
                wspace=0.2,cmap='RdPu',vmax='p100',ncols=5)
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1,1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(adata,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(adata,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 14 (unknown):
```unknown
number of hvgs: 3248
```

Example 15 (python):
```python
genes = ['TBX3','TBX18','SHOX2','HCN1','HCN4','CACNA1D', 'GJA1',# SANPC
         'GREB1L','GNG11','NRXN3','RSPO3', # AVNPC
         'PITX2','KCNMB2','PCDH7','RYR3','MBP', # aCM_left
         'NTM','ROR1','KIF26B','PLCL1','ANGPT1', # aCM_right
         'NKX2-5','MKI67','TOP2A'
        ]
genes = [x for x in genes if x in adata.var_names]
sc.pl.embedding(adata,basis='X_umap',color=genes,
               ncols=5, wspace=0.2,cmap='RdPu',vmax='p99.7')
```

Example 16 (unknown):
```unknown
resolution_sel = 1
```

Example 17 (python):
```python
markers = {
    'CM': ['TTN', 'TNNT2','MYH6',
        'MYH7', 'MYL2', 'FHL2', # vCN
            'NPPA', 'MYL7', 'MYL4'], #aCM
    'FB': ['DCN', 'GCN', 'PDGFRA','COL1A1','COL1A2'],
    'EC': ['VWF', 'PECAM1', 'CDH5','RGCC', 'FABP5',],
    'Peri': ['RGS5', 'ABCC9', 'KCNJ8'],
    'SMC': ['MYH11', 'TAGLN', 'ACTA2'],
    'Meso': ['MSLN', 'WT1', 'BNC1'],
    'Neuro': ['PLP1', 'NRXN1', 'NRXN3','PRPH', 'NEFL', 'NEFM', 'NEFH', 'STMN2', 'SST', 'SLC10A4', 'SLC18A3', 'TH', 'NPY'],
    'Adipo': ['GPAM', 'FASN', 'LEP'],
    'Myelo': ['CD14', 'C1QA', 'CD68','LYVE1','TIMD4'],
    'Lymph': ['CD8A', 'IL7R', 'CD40LG'],
    'B': ['CD19','SDC1','IGHG1','IGHA1'],
    'Mast': ['KIT', 'CPA3'],
}

for key in markers.keys():
    markers[key]=[x for x in markers[key] if x in adata.var_names]

sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              markers, 
              groupby=f'leiden_{str(resolution_sel)}',
              dendrogram=True,
              standard_scale="var",
              color_map="Reds",
              swap_axes=False,
             )
```

Example 18 (unknown):
```unknown
WARNING: Groups are not reordered because the `groupby` categories and the `var_group_labels` are different.
categories: 0, 1, 2, etc.
var_group_labels: CM, FB, EC, etc.
```

Example 19 (python):
```python
genes = ['TBX3','TBX18','SHOX2','HCN1','HCN4','CACNA1D', 'GJA1',# SANPC
         'GREB1L','GNG11','NRXN3','RSPO3', # AVNPC
         'PITX2','KCNMB2','PCDH7','RYR3','MBP', # aCM_left
         'NTM','ROR1','KIF26B','PLCL1','ANGPT1', # aCM_right
         'MKI67','TOP2A','NKX2-5'
        ]
genes = [x for x in genes if x in adata.var_names]

sc.tl.dendrogram(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.pl.dotplot(adata,
              var_names=genes,
              groupby=f'leiden_{str(resolution_sel)}',
              standard_scale='var',
              dendrogram=True
             )
```

Example 20 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True)
dp.add_totals().show()
```

Example 21 (unknown):
```unknown
[1.10352081 1.83785365 1.6291659  0.70335836 1.24040478]
```

Example 22 (python):
```python
# DEG analysis
print(adata.X.data[:5])
adata.uns['log1p']['base']=None
sc.tl.rank_genes_groups(adata,groupby=f'leiden_{str(resolution_sel)}')
sc.tl.dendrogram(adata, groupby=f'leiden_{str(resolution_sel)}')
dp=sc.pl.rank_genes_groups_dotplot(adata, dendrogram=True, standard_scale='var', return_fig=True, n_genes=5)
dp.add_totals().show()
```

Example 23 (unknown):
```unknown
[1.10352081 1.83785365 1.6291659  0.70335836 1.24040478]
```

Example 24 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=f'leiden_{str(resolution_sel)}',
                wspace=0.3, legend_loc='on data')
```

Example 25 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['celltypist_midmod2fine', 'conf_score_midmod2fine'],
                wspace=0.7)
```

Example 26 (python):
```python
sc.pl.violin(adata,keys=['total_counts','bin_count'],groupby=f'leiden_{str(resolution_sel)}')
```

Example 27 (unknown):
```unknown
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
Using categorical units to plot a list of strings that are all parsable as floats or dates. If these strings should be plotted as numbers, cast to the appropriate data type before plotting.
```

Example 28 (python):
```python
# annotate
adata.obs['fine_grain_manual'] = adata.obs[f'leiden_{str(resolution_sel)}'].copy()
adata.obs.replace({'fine_grain_manual':{
    '0':'AtrialCardiomyocytesLeft',
    '1':'unassigned',
    '2':'AtrialCardiomyocytesRight',
    '3':'AtrialCardiomyocytesRight',
    '4':'AtrioventricularNodePacemakerCells',
    '5':'SinoatrialNodePacemakerCells',
    '6':'AtrialCardiomyocytesRight',
    '7':'SinoatrialNodePacemakerCells',
    '8':'AtrialCardiomyocytesLeft',
}},inplace=True)
adata.obs['fine_grain_manual'].value_counts()
```

Example 29 (unknown):
```unknown
fine_grain_manual
AtrialCardiomyocytesRight             5060
AtrialCardiomyocytesLeft              3107
unassigned                            2625
AtrioventricularNodePacemakerCells    1602
SinoatrialNodePacemakerCells          1510
Name: count, dtype: int64
```

Example 30 (python):
```python
ad = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw_finegrain_updated.h5ad',backed='r')
set(ad.obs['fine_grain'])
```

Example 31 (unknown):
```unknown
{'AtrialCardiomyocytesCycling',
 'AtrialCardiomyocytesLeft',
 'AtrialCardiomyocytesRight',
 'AtrioventricularNodePacemakerCells',
 'BCells',
 'BCellsMS4A1pos',
 'ChromaffinCells',
 'CoronaryArterialEndothelialCells',
 'CoronaryCapillaryEndothelialCells',
 'CoronaryPericytes',
 'CoronarySmoothMuscleCells',
 'CoronaryVenousEndothelialCells',
 'CoronaryVesselAdventitialFibroblasts',
 'DendriticCellsMature',
 'DendriticCellsType1',
 'DuctusArteriosusSmoothMuscleCells',
 'EndocardialCells',
 'EndocardialCushionCells',
 'EpicardiumDerivedCells',
 'GreatVesselAdventitialFibroblasts',
 'GreatVesselArterialEndothelialCells',
 'GreatVesselSmoothMuscleCells',
 'GreatVesselVenousEndothelialCells',
 'InnateLymphoidCells',
 'LymphNodeFibroblasticReticularCells',
 'LymphaticEndothelialCells',
 'MacrophagesATF3pos',
 'MacrophagesCX3CR1pos',
 'MacrophagesLYVE1pos',
 'MacrophagesTIMD4pos',
 'MastCells',
 'Megakaryocytes',
 'MesothelialEpicardialCells',
 'MonocyteDerivedCells',
 'Monocytes',
 'MonocytesMPOpos',
 'MyocardialInterstitialFibroblasts',
 'Myofibroblasts',
 'NaturalKillerCells',
 'NeuronPrecursors',
 'ParasympatheticNeurons',
 'PericardialCellsFibrous',
 'PericardialCellsIntermediate',
 'PericardialCellsParietal',
 'PlasmacytoidDendriticCells',
 'ProBCells',
 'SchwannCellPrecursors',
 'SchwannCells',
 'SinoatrialNodePacemakerCells',
 'SubEpicardialFibroblasts',
 'SympatheticNeurons',
 'TCellsCD4pos',
 'TCellsCD8pos',
 'TregsCD4pos',
 'ValveEndothelialCells',
 'ValveInterstitialCells',
 'VentricularCardiomyocytesCycling',
 'VentricularCardiomyocytesLeftCompact',
 'VentricularCardiomyocytesLeftTrabeculated',
 'VentricularCardiomyocytesRightCompact',
 'VentricularCardiomyocytesRightTrabeculated',
 'VentricularConductionSystemDistal',
 'VentricularConductionSystemProximal'}
```

Example 32 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',
                frameon=False,
                wspace=0.3)
```

Example 33 (python):
```python
genes = ['TBX3','TBX18','SHOX2','HCN1','HCN4','CACNA1D', 'GJA1',# SANPC
         'GREB1L','GNG11','NRXN3','RSPO3', # AVNPC
         'PITX2','KCNMB2','PCDH7','RYR3','MBP', # aCM_left
         'NTM','ROR1','KIF26B','PLCL1','ANGPT1', # aCM_right
         'MKI67','TOP2A','NKX2-5'
        ]
genes = [x for x in genes if x in adata.var_names]

sc.tl.dendrogram(adata,groupby='fine_grain_manual')
sc.pl.dotplot(adata,
              var_names=genes,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 34 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color=['SHOX2','fine_grain_manual'],
                frameon=False, cmap='RdPu', size=15,
                wspace=0.3, show=False)
plt.savefig(f'{figdir}/VisiumHD_euploid_atrialCM_UMAP.pdf', bbox_inches="tight",dpi=300)
```

Example 35 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 39 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'H', 'L', 'N', 'O', 'P', 'R', 'S', 'X', 'a', 'c', 'd', 'e', 'f', 'four', 'g', 'h', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'r', 's', 'space', 't', 'three', 'two', 'u', 'underscore', 'v', 'y', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 20, 21, 22, 23, 36, 38, 43, 47, 49, 50, 51, 53, 54, 59, 66, 68, 70, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92]
Closed glyph list over 'MATH': 39 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'H', 'L', 'N', 'O', 'P', 'R', 'S', 'X', 'a', 'c', 'd', 'e', 'f', 'four', 'g', 'h', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'r', 's', 'space', 't', 'three', 'two', 'u', 'underscore', 'v', 'y', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 20, 21, 22, 23, 36, 38, 43, 47, 49, 50, 51, 53, 54, 59, 66, 68, 70, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92]
Closing glyph list over 'GSUB': 39 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'H', 'L', 'N', 'O', 'P', 'R', 'S', 'X', 'a', 'c', 'd', 'e', 'f', 'four', 'g', 'h', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'r', 's', 'space', 't', 'three', 'two', 'u', 'underscore', 'v', 'y', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 20, 21, 22, 23, 36, 38, 43, 47, 49, 50, 51, 53, 54, 59, 66, 68, 70, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92]
Closed glyph list over 'GSUB': 44 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'H', 'L', 'N', 'O', 'P', 'R', 'S', 'X', 'a', 'c', 'd', 'e', 'f', 'fi', 'fl', 'four', 'g', 'h', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'r', 's', 'space', 't', 'three', 'two', 'u', 'underscore', 'uniFB00', 'uniFB03', 'uniFB04', 'v', 'y', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 20, 21, 22, 23, 36, 38, 43, 47, 49, 50, 51, 53, 54, 59, 66, 68, 70, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92, 5038, 5039, 5040, 5041, 5042]
Closing glyph list over 'glyf': 44 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'C', 'H', 'L', 'N', 'O', 'P', 'R', 'S', 'X', 'a', 'c', 'd', 'e', 'f', 'fi', 'fl', 'four', 'g', 'h', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'r', 's', 'space', 't', 'three', 'two', 'u', 'underscore', 'uniFB00', 'uniFB03', 'uniFB04', 'v', 'y', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 20, 21, 22, 23, 36, 38, 43, 47, 49, 50, 51, 53, 54, 59, 66, 68, 70, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92, 5038, 5039, 5040, 5041, 5042]
Closed glyph list over 'glyf': 44 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'C', 'H', 'L', 'N', 'O', 'P', 'R', 'S', 'X', 'a', 'c', 'd', 'e', 'f', 'fi', 'fl', 'four', 'g', 'h', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'r', 's', 'space', 't', 'three', 'two', 'u', 'underscore', 'uniFB00', 'uniFB03', 'uniFB04', 'v', 'y', 'zero']
Glyph IDs:   [0, 1, 2, 3, 19, 20, 21, 22, 23, 36, 38, 43, 47, 49, 50, 51, 53, 54, 59, 66, 68, 70, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88, 89, 92, 5038, 5039, 5040, 5041, 5042]
Retaining 44 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 36 (python):
```python
sc.pl.embedding(adata,basis='X_umap',
                color='fine_grain_manual',groups='SinoatrialNodePacemakerCells',
                frameon=False,
                wspace=0.3)
```

Example 37 (python):
```python
san = adata[adata.obs['fine_grain_manual']=='SinoatrialNodePacemakerCells']

# drop leiden clustering results
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    san.obs.drop(f'leiden_{str(resolution)}',axis=1,inplace=True)
san
```

Example 38 (unknown):
```unknown
AnnData object with n_obs × n_vars = 1510 × 17650
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'dendrogram_fine_grain_manual', 'dendrogram_leiden_1', 'dendrogram_leiden_1.5', 'fine_grain_manual_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1_colors', 'leiden_2', 'log1p', 'neighbors', 'pca', 'rank_genes_groups', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 39 (unknown):
```unknown
# remove mid_grain markers
mask = [x not in list(midgrain_markers['names']) for x in san.var_names]
bdata = san[:,mask]
bdata
```

Example 40 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 1510 × 16391
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'dendrogram_fine_grain_manual', 'dendrogram_leiden_1', 'dendrogram_leiden_1.5', 'fine_grain_manual_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1_colors', 'leiden_2', 'log1p', 'neighbors', 'pca', 'rank_genes_groups', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 41 (python):
```python
# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
san.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(san,n_neighbors=15,use_rep='X_pca')
```

Example 42 (unknown):
```unknown
number of hvgs: 3692
```

Example 43 (python):
```python
sc.tl.umap(san,min_dist=0.1)
```

Example 44 (python):
```python
%%time
# reclustering
# leiden clustering
resolutions_list = [0.2,0.4,0.6,1, 1.2,1.5,2]
for resolution in resolutions_list:
    sc.tl.leiden(san,resolution=resolution,key_added=f'leiden_{str(resolution)}',n_iterations=2)
sc.pl.embedding(san,basis='X_umap',
                color=[f'leiden_{str(resolution)}' for resolution in resolutions_list],
                wspace=0.3)
```

Example 45 (unknown):
```unknown
CPU times: user 1.45 s, sys: 31.3 ms, total: 1.48 s
Wall time: 1.95 s
```

Example 46 (unknown):
```unknown
san_resolution_sel = 0.6
```

Example 47 (python):
```python
genes = ['SHOX2','HCN4','TBX3','TBX18',
         'GJA1', # CX43
         'GJA5', # CX40
         'NKX2-5',
         'NPPA'
        ]
genes = [x for x in genes if x in san.var_names]
sc.pl.embedding(san,basis='X_umap',
                color=[f'leiden_{str(san_resolution_sel)}']+genes,
               ncols=5, wspace=0.15,cmap='RdPu',vmax='p100')
```

Example 48 (unknown):
```unknown
# annotate
san.obs['fine_grain_manual'] = san.obs[f'leiden_{str(san_resolution_sel)}'].copy()
san.obs.replace({'fine_grain_manual':{
    '0':'SinoatrialNodePacemakerCellsHead',
    '1':'SinoatrialNodePacemakerCellsHorn',
    '2':'SinoatrialNodePacemakerCellsTail',
}},inplace=True)
san.obs['fine_grain_manual'].value_counts()
```

Example 49 (unknown):
```unknown
fine_grain_manual
SinoatrialNodePacemakerCellsHead    636
SinoatrialNodePacemakerCellsHorn    501
SinoatrialNodePacemakerCellsTail    373
Name: count, dtype: int64
```

Example 50 (unknown):
```unknown
san.obs['fine_grain_manual'] = san.obs['fine_grain_manual'].astype('category')
san.obs['fine_grain_manual'] = san.obs['fine_grain_manual'].cat.reorder_categories([
    'SinoatrialNodePacemakerCellsHorn',
    'SinoatrialNodePacemakerCellsHead',
    'SinoatrialNodePacemakerCellsTail'
])
```

Example 51 (python):
```python
genes = ['TBX3','TBX18',
         'GJA1', # CX43
         'GJA5', # CX40
         'NKX2-5','NPPA']
genes = [x for x in genes if x in san.var_names]
sc.pl.dotplot(san,
              var_names=genes,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=False
             )
```

Example 52 (python):
```python
res = utils.rank_genes_groups_with_mean(san,groupby='fine_grain_manual',
                                        log2fc_min=0,method='t-test')
n_rank = 20
markers = {}
for group in res['group'].unique():
    res_sub = res[res['group']==group]
    res_sub = res_sub.sort_values('scores',ascending=False)
    markers[group] = list(res_sub['names'])[:n_rank]

sc.tl.dendrogram(san,groupby='fine_grain_manual',var_names=markers,)
sc.pl.dotplot(san,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 53 (python):
```python
# with removing mid_grain markers
# remove mid_grain markers
mask = [x not in list(midgrain_markers['names']) for x in san.var_names]
bdata = san[:,mask]

res = utils.rank_genes_groups_with_mean(bdata,groupby='fine_grain_manual',
                                        log2fc_min=0,method='t-test')
n_rank = 20
markers = {}
for group in res['group'].unique():
    res_sub = res[res['group']==group]
    res_sub = res_sub.sort_values('scores',ascending=False)
    markers[group] = list(res_sub['names'])[:n_rank]

sc.tl.dendrogram(bdata,groupby='fine_grain_manual',var_names=markers,)
sc.pl.dotplot(bdata,
              var_names=markers,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True
             )
```

Example 54 (python):
```python
# add the annotation to the original adata
adata.obs['fine_grain_manual'] = adata.obs['fine_grain_manual'].astype('str')
adata.obs.loc[san.obs_names,'fine_grain_manual'] = san.obs['fine_grain_manual'].astype('str').copy()
adata.obs['fine_grain_manual'].value_counts()
```

Example 55 (unknown):
```unknown
fine_grain_manual
AtrialCardiomyocytesRight             5060
AtrialCardiomyocytesLeft              3107
unassigned                            2625
AtrioventricularNodePacemakerCells    1602
SinoatrialNodePacemakerCellsHead       636
SinoatrialNodePacemakerCellsHorn       501
SinoatrialNodePacemakerCellsTail       373
Name: count, dtype: int64
```

Example 56 (python):
```python
# save
adata.write(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
print(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
```

Example 57 (unknown):
```unknown
... storing 'fine_grain_manual' as categorical
```

Example 58 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/subsets/HEA_FOET14880396_AtrialCardiomyocytes_b2c_cells_filtered_lognorm.h5ad
```

Example 59 (python):
```python
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_{celltype}_b2c_cells_filtered_lognorm.h5ad')
adata
```

Example 60 (unknown):
```unknown
AnnData object with n_obs × n_vars = 13904 × 17650
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'dendrogram_fine_grain_manual', 'dendrogram_leiden_1', 'dendrogram_leiden_1.5', 'fine_grain_manual_colors', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_0.6_colors', 'leiden_1', 'leiden_1.2', 'leiden_1.2_colors', 'leiden_1.5', 'leiden_1.5_colors', 'leiden_1_colors', 'leiden_2', 'leiden_2_colors', 'log1p', 'neighbors', 'pca', 'rank_genes_groups', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 61 (python):
```python
set(adata.obs['fine_grain_manual'])
```

Example 62 (unknown):
```unknown
{'AtrialCardiomyocytesLeft',
 'AtrialCardiomyocytesRight',
 'AtrioventricularNodePacemakerCells',
 'SinoatrialNodePacemakerCellsHead',
 'SinoatrialNodePacemakerCellsHorn',
 'SinoatrialNodePacemakerCellsTail',
 'unassigned'}
```

Example 63 (python):
```python
san = adata[adata.obs['fine_grain_manual'].isin(['SinoatrialNodePacemakerCellsHead',
 'SinoatrialNodePacemakerCellsHorn',
 'SinoatrialNodePacemakerCellsTail'])]
san
```

Example 64 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 1510 × 17650
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'celltypist_coarse', 'conf_score_coarse', 'celltypist_coarse_fin', 'celltypist_coarse2midmod', 'conf_score_coarse2midmod', 'celltypist_mid_fin', 'celltypist_midmod2fine', 'conf_score_midmod2fine', 'leiden_0.2', 'leiden_0.4', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_2', 'fine_grain_manual'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm'
    uns: 'celltypist_coarse2midmod_colors', 'celltypist_coarse_colors', 'celltypist_coarse_fin_colors', 'celltypist_mid_fin_colors', 'celltypist_midmod2fine_colors', 'dendrogram_fine_grain_manual', 'dendrogram_leiden_1', 'dendrogram_leiden_1.5', 'hvg', 'leiden_0.2', 'leiden_0.2_colors', 'leiden_0.4', 'leiden_0.4_colors', 'leiden_0.6', 'leiden_1', 'leiden_1.2', 'leiden_1.5', 'leiden_1_colors', 'leiden_2', 'log1p', 'neighbors', 'pca', 'rank_genes_groups', 'spatial', 'umap'
    obsm: 'X_pca', 'X_umap', 'spatial', 'spatial_cropped_150_buffer'
    varm: 'PCs'
    obsp: 'connectivities', 'distances'
```

Example 65 (python):
```python
# remove mid_grain markers
mask = [x not in list(midgrain_markers['names']) for x in san.var_names]
bdata = san[:,mask]

# HVGs, using the object which doesn't have midgrain markers
sc.pp.highly_variable_genes(bdata, flavor='seurat', min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'number of hvgs: {sum(bdata.var.highly_variable)}')
# PCA using the degected HVGs
sc.tl.pca(bdata, use_highly_variable=True)
san.obsm['X_pca'] = bdata.obsm['X_pca'].copy()
# create neighbour graph
sc.pp.neighbors(san,n_neighbors=15,use_rep='X_pca')
# umap
sc.tl.umap(san,min_dist=0.1)
```

Example 66 (unknown):
```unknown
number of hvgs: 3692
```

Example 67 (python):
```python
sc.pl.embedding(san,basis='X_umap',
                color=['fine_grain_manual'],
               frameon=False,show=False)
plt.savefig(f'{figdir}/VisiumHD_euploid_SANPC_UMAP.pdf', bbox_inches="tight",dpi=300)
```

Example 68 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 27 glyphs before
Glyph names: ['.notdef', '.null', 'C', 'H', 'N', 'P', 'S', 'T', 'a', 'c', 'd', 'e', 'f', 'g', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'underscore']
Glyph IDs:   [0, 1, 2, 3, 38, 43, 49, 51, 54, 55, 66, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88]
Closed glyph list over 'MATH': 27 glyphs after
Glyph names: ['.notdef', '.null', 'C', 'H', 'N', 'P', 'S', 'T', 'a', 'c', 'd', 'e', 'f', 'g', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'underscore']
Glyph IDs:   [0, 1, 2, 3, 38, 43, 49, 51, 54, 55, 66, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88]
Closing glyph list over 'GSUB': 27 glyphs before
Glyph names: ['.notdef', '.null', 'C', 'H', 'N', 'P', 'S', 'T', 'a', 'c', 'd', 'e', 'f', 'g', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'underscore']
Glyph IDs:   [0, 1, 2, 3, 38, 43, 49, 51, 54, 55, 66, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88]
Closed glyph list over 'GSUB': 32 glyphs after
Glyph names: ['.notdef', '.null', 'C', 'H', 'N', 'P', 'S', 'T', 'a', 'c', 'd', 'e', 'f', 'fi', 'fl', 'g', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'underscore', 'uniFB00', 'uniFB03', 'uniFB04']
Glyph IDs:   [0, 1, 2, 3, 38, 43, 49, 51, 54, 55, 66, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88, 5038, 5039, 5040, 5041, 5042]
Closing glyph list over 'glyf': 32 glyphs before
Glyph names: ['.notdef', '.null', 'C', 'H', 'N', 'P', 'S', 'T', 'a', 'c', 'd', 'e', 'f', 'fi', 'fl', 'g', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'underscore', 'uniFB00', 'uniFB03', 'uniFB04']
Glyph IDs:   [0, 1, 2, 3, 38, 43, 49, 51, 54, 55, 66, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88, 5038, 5039, 5040, 5041, 5042]
Closed glyph list over 'glyf': 32 glyphs after
Glyph names: ['.notdef', '.null', 'C', 'H', 'N', 'P', 'S', 'T', 'a', 'c', 'd', 'e', 'f', 'fi', 'fl', 'g', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'r', 's', 'space', 't', 'u', 'underscore', 'uniFB00', 'uniFB03', 'uniFB04']
Glyph IDs:   [0, 1, 2, 3, 38, 43, 49, 51, 54, 55, 66, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 85, 86, 87, 88, 5038, 5039, 5040, 5041, 5042]
Retaining 32 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

Example 69 (python):
```python
?sc.pl.dotplot
```

Example 70 (python):
```python
Signature:
sc.pl.dotplot(
    adata: 'AnnData',
    var_names: '_VarNames | Mapping[str, _VarNames]',
    groupby: 'str | Sequence[str]',
    *,
    use_raw: 'bool | None' = None,
    log: 'bool' = False,
    num_categories: 'int' = 7,
    expression_cutoff: 'float' = 0.0,
    mean_only_expressed: 'bool' = False,
    cmap: 'str' = 'Reds',
    dot_max: 'float | None' = None,
    dot_min: 'float | None' = None,
    standard_scale: "Literal['var', 'group'] | None" = None,
    smallest_dot: 'float | None' = 0.0,
    title: 'str | None' = None,
    colorbar_title: 'str | None' = 'Mean expression\nin group',
    size_title: 'str | None' = 'Fraction of cells\nin group (%)',
    figsize: 'tuple[float, float] | None' = None,
    dendrogram: 'bool | str' = False,
    gene_symbols: 'str | None' = None,
    var_group_positions: 'Sequence[tuple[int, int]] | None' = None,
    var_group_labels: 'Sequence[str] | None' = None,
    var_group_rotation: 'float | None' = None,
    layer: 'str | None' = None,
    swap_axes: 'bool | None' = False,
    dot_color_df: 'pd.DataFrame | None' = None,
    show: 'bool | None' = None,
    save: 'str | bool | None' = None,
    ax: '_AxesSubplot | None' = None,
    return_fig: 'bool | None' = False,
    vmin: 'float | None' = None,
    vmax: 'float | None' = None,
    vcenter: 'float | None' = None,
    norm: 'Normalize | None' = None,
    **kwds,
) -> 'DotPlot | dict | None'
Docstring:
Makes a *dot plot* of the expression values of `var_names`.

For each var_name and each `groupby` category a dot is plotted.
Each dot represents two values: mean expression within each category
(visualized by color) and fraction of cells expressing the `var_name` in the
category (visualized by the size of the dot). If `groupby` is not given,
the dotplot assumes that all data belongs to a single category.

.. note::
   A gene is considered expressed if the expression value in the `adata` (or
   `adata.raw`) is above the specified threshold which is zero by default.

An example of dotplot usage is to visualize, for multiple marker genes,
the mean value and the percentage of cells expressing the gene
across  multiple clusters.

This function provides a convenient interface to the :class:`~scanpy.pl.DotPlot`
class. If you need more flexibility, you should use :class:`~scanpy.pl.DotPlot`
directly.

Parameters
----------
adata : 'AnnData'
    Annotated data matrix.
var_names : '_VarNames | Mapping[str, _VarNames]'
    `var_names` should be a valid subset of `adata.var_names`.
    If `var_names` is a mapping, then the key is used as label
    to group the values (see `var_group_labels`). The mapping values
    should be sequences of valid `adata.var_names`. In this
    case either coloring or 'brackets' are used for the grouping
    of var names depending on the plot. When `var_names` is a mapping,
    then the `var_group_labels` and `var_group_positions` are set.
groupby : 'str | Sequence[str]'
    The key of the observation grouping to consider.
use_raw : 'bool | None', optional (default: None)
    Use `raw` attribute of `adata` if present.
log : 'bool', optional (default: False)
    Plot on logarithmic axis.
num_categories : 'int', optional (default: 7)
    Only used if groupby observation is not categorical. This value
    determines the number of groups into which the groupby observation
    should be subdivided.
categories_order
    Order in which to show the categories. Note: add_dendrogram or add_totals
    can change the categories order.
figsize : 'tuple[float, float] | None', optional (default: None)
    Figure size when `multi_panel=True`.
    Otherwise the `rcParam['figure.figsize]` value is used.
    Format is (width, height)
dendrogram : 'bool | str', optional (default: False)
    If True or a valid dendrogram key, a dendrogram based on the hierarchical
    clustering between the `groupby` categories is added.
    The dendrogram information is computed using :func:`scanpy.tl.dendrogram`.
    If `tl.dendrogram` has not been called previously the function is called
    with default parameters.
gene_symbols : 'str | None', optional (default: None)
    Column name in `.var` DataFrame that stores gene symbols.
    By default `var_names` refer to the index column of the `.var` DataFrame.
    Setting this option allows alternative names to be used.
var_group_positions : 'Sequence[tuple[int, int]] | None', optional (default: None)
    Use this parameter to highlight groups of `var_names`.
    This will draw a 'bracket' or a color block between the given start and end
    positions. If the parameter `var_group_labels` is set, the corresponding
    labels are added on top/left. E.g. `var_group_positions=[(4,10)]`
    will add a bracket between the fourth `var_name` and the tenth `var_name`.
    By giving more positions, more brackets/color blocks are drawn.
var_group_labels : 'Sequence[str] | None', optional (default: None)
    Labels for each of the `var_group_positions` that want to be highlighted.
var_group_rotation : 'float | None', optional (default: None)
    Label rotation degrees.
    By default, labels larger than 4 characters are rotated 90 degrees.
layer : 'str | None', optional (default: None)
    Name of the AnnData object layer that wants to be plotted. By default adata.raw.X is plotted.
    If `use_raw=False` is set, then `adata.X` is plotted. If `layer` is set to a valid layer name,
    then the layer is plotted. `layer` takes precedence over `use_raw`.
title : 'str | None', optional (default: None)
    Title for the figure
colorbar_title : 'str | None', optional (default: 'Mean expression\nin group')
    Title for the color bar. New line character (\n) can be used.
cmap : 'str', optional (default: 'Reds')
    String denoting matplotlib color map.
standard_scale : "Literal['var', 'group'] | None", optional (default: None)
    Whether or not to standardize the given dimension between 0 and 1, meaning for
    each variable or group, subtract the minimum and divide each by its maximum.
swap_axes : 'bool | None', optional (default: False)
     By default, the x axis contains `var_names` (e.g. genes) and the y axis
     the `groupby` categories. By setting `swap_axes` then x are the
     `groupby` categories and y the `var_names`.
return_fig : 'bool | None', optional (default: False)
    Returns :class:`DotPlot` object. Useful for fine-tuning
    the plot. Takes precedence over `show=False`.

size_title : 'str | None', optional (default: 'Fraction of cells\nin group (%)')
    Title for the size legend. New line character (\n) can be used.
expression_cutoff : 'float', optional (default: 0.0)
    Expression cutoff that is used for binarizing the gene expression and
    determining the fraction of cells expressing given genes. A gene is
    expressed only if the expression value is greater than this threshold.
mean_only_expressed : 'bool', optional (default: False)
    If True, gene expression is averaged only over the cells
    expressing the given genes.
dot_max : 'float | None', optional (default: None)
    If none, the maximum dot size is set to the maximum fraction value found
    (e.g. 0.6). If given, the value should be a number between 0 and 1.
    All fractions larger than dot_max are clipped to this value.
dot_min : 'float | None', optional (default: None)
    If none, the minimum dot size is set to 0. If given,
    the value should be a number between 0 and 1.
    All fractions smaller than dot_min are clipped to this value.
smallest_dot : 'float | None', optional (default: 0.0)
    If none, the smallest dot has size 0.
    All expression levels with `dot_min` are plotted with this size.
show : 'bool | None', optional (default: None)
     Show the plot, do not return axis.
save : 'str | bool | None', optional (default: None)
    If `True` or a `str`, save the figure.
    A string is appended to the default filename.
    Infer the filetype if ending on {`'.pdf'`, `'.png'`, `'.svg'`}.
ax : '_AxesSubplot | None', optional (default: None)
    A matplotlib axes object. Only works if plotting a single component.
vmin : 'float | None', optional (default: None)
    The value representing the lower limit of the color scale. Values smaller than vmin are plotted
    with the same color as vmin.
vmax : 'float | None', optional (default: None)
    The value representing the upper limit of the color scale. Values larger than vmax are plotted
    with the same color as vmax.
vcenter : 'float | None', optional (default: None)
    The value representing the center of the color scale. Useful for diverging colormaps.
norm : 'Normalize | None', optional (default: None)
    Custom color normalization object from matplotlib. See
    `https://matplotlib.org/stable/tutorials/colors/colormapnorms.html` for details.
kwds : _empty
    Are passed to :func:`matplotlib.pyplot.scatter`.

Returns
-------
If `return_fig` is `True`, returns a :class:`~scanpy.pl.DotPlot` object,
else if `show` is false, return axes dict

See also
--------
:class:`~scanpy.pl.DotPlot`: The DotPlot class can be used to to control
    several visual parameters not available in this function.
:func:`~scanpy.pl.rank_genes_groups_dotplot`: to plot marker genes
    identified using the :func:`~scanpy.tl.rank_genes_groups` function.

Examples
--------

Create a dot plot using the given markers and the PBMC example dataset grouped by
the category 'bulk_labels'.

.. plot::
    :context: close-figs

    import scanpy as sc
    adata = sc.datasets.pbmc68k_reduced()
    markers = ['C1QA', 'PSAP', 'CD79A', 'CD79B', 'CST3', 'LYZ']
    sc.pl.dotplot(adata, markers, groupby='bulk_labels', dendrogram=True)

Using var_names as dict:

.. plot::
    :context: close-figs

    markers = {'T-cell': 'CD3D', 'B-cell': 'CD79A', 'myeloid': 'CST3'}
    sc.pl.dotplot(adata, markers, groupby='bulk_labels', dendrogram=True)

Get DotPlot object for fine tuning

.. plot::
    :context: close-figs

    dp = sc.pl.dotplot(adata, markers, 'bulk_labels', return_fig=True)
    dp.add_totals().style(dot_edge_color='black', dot_edge_lw=0.5).show()

The axes used can be obtained using the get_axes() method

.. code-block:: python

    axes_dict = dp.get_axes()
    print(axes_dict)
File:      /software/cellgen/team205/kk18/envs/generic_env/lib/python3.9/site-packages/scanpy/plotting/_dotplot.py
Type:      function
```

Example 71 (python):
```python
genes = ['TBX3','TBX18',
         'GJA1', # CX43
         'GJA5', # CX40
         'NKX2-5']
genes = [x for x in genes if x in san.var_names]
sc.tl.dendrogram(san,groupby='fine_grain_manual')
sc.pl.dotplot(san,
              var_names=genes,
              groupby='fine_grain_manual',
              standard_scale='var',
              dendrogram=True,show=False
             )
plt.savefig(f'{figdir}/VisiumHD_euploid_SANPC_dotplot-markers.pdf', bbox_inches="tight",dpi=300)
```

Example 72 (unknown):
```unknown
maxp pruned
cmap pruned
kern dropped
post pruned
FFTM dropped
GPOS pruned
GSUB pruned
glyf pruned
Added gid0 to subset
Added first four glyphs to subset
Closing glyph list over 'MATH': 47 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'F', 'G', 'H', 'J', 'K', 'M', 'N', 'P', 'S', 'T', 'X', 'a', 'c', 'd', 'e', 'eight', 'f', 'five', 'g', 'hyphen', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'parenleft', 'parenright', 'percent', 'r', 's', 'seven', 'space', 't', 'three', 'two', 'u', 'x', 'zero']
Glyph IDs:   [0, 1, 2, 3, 8, 11, 12, 16, 19, 20, 21, 22, 24, 26, 27, 36, 37, 38, 41, 42, 43, 45, 46, 48, 49, 51, 54, 55, 59, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 83, 85, 86, 87, 88, 91]
Closed glyph list over 'MATH': 53 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'F', 'G', 'H', 'J', 'K', 'M', 'N', 'P', 'S', 'T', 'X', 'a', 'c', 'd', 'e', 'eight', 'f', 'five', 'g', 'hyphen', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'parenleft', 'parenright', 'percent', 'r', 's', 'seven', 'space', 't', 'three', 'two', 'u', 'uni239B', 'uni239C', 'uni239D', 'uni239E', 'uni239F', 'uni23A0', 'x', 'zero']
Glyph IDs:   [0, 1, 2, 3, 8, 11, 12, 16, 19, 20, 21, 22, 24, 26, 27, 36, 37, 38, 41, 42, 43, 45, 46, 48, 49, 51, 54, 55, 59, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 83, 85, 86, 87, 88, 91, 3506, 3507, 3508, 3509, 3510, 3511]
Closing glyph list over 'GSUB': 53 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'F', 'G', 'H', 'J', 'K', 'M', 'N', 'P', 'S', 'T', 'X', 'a', 'c', 'd', 'e', 'eight', 'f', 'five', 'g', 'hyphen', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'parenleft', 'parenright', 'percent', 'r', 's', 'seven', 'space', 't', 'three', 'two', 'u', 'uni239B', 'uni239C', 'uni239D', 'uni239E', 'uni239F', 'uni23A0', 'x', 'zero']
Glyph IDs:   [0, 1, 2, 3, 8, 11, 12, 16, 19, 20, 21, 22, 24, 26, 27, 36, 37, 38, 41, 42, 43, 45, 46, 48, 49, 51, 54, 55, 59, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 83, 85, 86, 87, 88, 91, 3506, 3507, 3508, 3509, 3510, 3511]
Closed glyph list over 'GSUB': 58 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'F', 'G', 'H', 'J', 'K', 'M', 'N', 'P', 'S', 'T', 'X', 'a', 'c', 'd', 'e', 'eight', 'f', 'fi', 'five', 'fl', 'g', 'hyphen', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'parenleft', 'parenright', 'percent', 'r', 's', 'seven', 'space', 't', 'three', 'two', 'u', 'uni239B', 'uni239C', 'uni239D', 'uni239E', 'uni239F', 'uni23A0', 'uniFB00', 'uniFB03', 'uniFB04', 'x', 'zero']
Glyph IDs:   [0, 1, 2, 3, 8, 11, 12, 16, 19, 20, 21, 22, 24, 26, 27, 36, 37, 38, 41, 42, 43, 45, 46, 48, 49, 51, 54, 55, 59, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 83, 85, 86, 87, 88, 91, 3506, 3507, 3508, 3509, 3510, 3511, 5038, 5039, 5040, 5041, 5042]
Closing glyph list over 'glyf': 58 glyphs before
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'F', 'G', 'H', 'J', 'K', 'M', 'N', 'P', 'S', 'T', 'X', 'a', 'c', 'd', 'e', 'eight', 'f', 'fi', 'five', 'fl', 'g', 'hyphen', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'parenleft', 'parenright', 'percent', 'r', 's', 'seven', 'space', 't', 'three', 'two', 'u', 'uni239B', 'uni239C', 'uni239D', 'uni239E', 'uni239F', 'uni23A0', 'uniFB00', 'uniFB03', 'uniFB04', 'x', 'zero']
Glyph IDs:   [0, 1, 2, 3, 8, 11, 12, 16, 19, 20, 21, 22, 24, 26, 27, 36, 37, 38, 41, 42, 43, 45, 46, 48, 49, 51, 54, 55, 59, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 83, 85, 86, 87, 88, 91, 3506, 3507, 3508, 3509, 3510, 3511, 5038, 5039, 5040, 5041, 5042]
Closed glyph list over 'glyf': 58 glyphs after
Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'F', 'G', 'H', 'J', 'K', 'M', 'N', 'P', 'S', 'T', 'X', 'a', 'c', 'd', 'e', 'eight', 'f', 'fi', 'five', 'fl', 'g', 'hyphen', 'i', 'k', 'l', 'm', 'n', 'nonmarkingreturn', 'o', 'one', 'p', 'parenleft', 'parenright', 'percent', 'r', 's', 'seven', 'space', 't', 'three', 'two', 'u', 'uni239B', 'uni239C', 'uni239D', 'uni239E', 'uni239F', 'uni23A0', 'uniFB00', 'uniFB03', 'uniFB04', 'x', 'zero']
Glyph IDs:   [0, 1, 2, 3, 8, 11, 12, 16, 19, 20, 21, 22, 24, 26, 27, 36, 37, 38, 41, 42, 43, 45, 46, 48, 49, 51, 54, 55, 59, 68, 70, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 83, 85, 86, 87, 88, 91, 3506, 3507, 3508, 3509, 3510, 3511, 5038, 5039, 5040, 5041, 5042]
Retaining 58 glyphs
head subsetting not needed
hhea subsetting not needed
maxp subsetting not needed
OS/2 subsetting not needed
hmtx subsetted
cmap subsetted
fpgm subsetting not needed
prep subsetting not needed
cvt  subsetting not needed
loca subsetting not needed
post subsetted
gasp subsetting not needed
MATH subsetted
GDEF subsetted
GPOS subsetted
GSUB subsetted
name subsetting not needed
glyf subsetted
head pruned
OS/2 Unicode ranges pruned: [0]
OS/2 CodePage ranges pruned: [0]
glyf pruned
GDEF pruned
GPOS pruned
GSUB pruned
name pruned
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/Visium_mapping/LvR_aCM_immature_CM_UMAP.html

**Contents:**
- Aims¶
- Read in data with all features¶
- ¶
- Split into GEX and ATAC objects¶
- Make AUC objects¶
- Find features which are Left or Right specific¶
- GEX¶
  - Functional interpretation of Left and Right CM sigantures¶
- ATAC¶
- AUCgene¶
- AUCregion¶
  - Plot features which are specific to Left or Right aCMs¶
- Find features which increase over time¶
  - Functional interpretation of genes which increase over time across all aCMs¶
- Find features which are associated with time and are increasing over time¶
- Fit expression of features which change (increase or decrease) over time¶
- GEX¶
- ATAC¶
- AUCgene¶
- AUCregion¶

We want to demonstrate:

LvR: features which are different at all time (DE testing)

L: features which are associated and increase over time L: features which are associated and decrease over time

R: features which are associated and increase over time R: features which are associated and decrease over time

We start by definign a stricter set of L and R specific genes

**Examples:**

Example 1 (unknown):
```unknown
jctk_dir="/lustre/scratch126/cellgen/team205/jc48/jupyter/jctk"
```

Example 2 (python):
```python
import importlib.util
import sys
import types
import inspect

# ensure the `jctk_dir` path is already specified

def import_and_list_functions(module_name, file_path):
    spec = importlib.util.spec_from_file_location(module_name, file_path)
    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module
    spec.loader.exec_module(module)
    
    function_names = []
    for name, obj in inspect.getmembers(module):
        if isinstance(obj, types.FunctionType):
            function_names.append(name)
    
    print(f"Imported from {module_name}: {function_names}")
    
    return module  # Return the module object

# Import and list functions, then assign to a variable
plot = import_and_list_functions("plot", f"{jctk_dir}/jctk/plot.py")
utils = import_and_list_functions("utils", f"{jctk_dir}/jctk/utils.py")
api = import_and_list_functions("api", f"{jctk_dir}/jctk/api.py")
```

Example 3 (python):
```python
Imported from plot: ['GO_plot', 'highlight_all_categories', 'highlight_categories', 'normalized_stacked_bar_chart']
Imported from utils: ['check_update', 'compute_median_count', 'downsample_adata_proportionately', 'downsample_adata_randomly', 'lognorm_to_counts_with_progress', 'rank_genes_vs_nearest', 'read_gsheet', 'remove_small_batches', 'rename_adata_var_index', 'rules_based_annotation', 'shapiro', 'test_normality']
Imported from api: ['add_genomic_coordinates', 'getClinVargenes', 'getHGNCgroup', 'getHGNCgroup_dict']
```

Example 4 (python):
```python
import anndata

def split_adata_by_modality(adata, modality_column, modality_values):
    """
    Splits an AnnData object into multiple AnnData objects based on a given modality column.
    
    Parameters:
        adata (anndata.AnnData): The original AnnData object.
        modality_column (str): The name of the column in adata.var that contains the modality information.
        modality_values (list): List of unique modalities that you want to split the adata into.
        
    Returns:
        dict: A dictionary where the keys are the modality names and the values are the new AnnData objects.
    """
    # Initialize an empty dictionary to hold the new AnnData objects
    splitted_adatas = {}
    
    # Loop through each modality value to create a new AnnData object
    for modality in modality_values:
        # Get the indices of the features belonging to the current modality
        feature_idx = adata.var[modality_column] == modality
        
        # Create a new AnnData object with only the features belonging to the current modality
        new_adata = anndata.AnnData(X=adata[:, feature_idx].X,
                                    obs=adata.obs,
                                    var=adata.var.loc[feature_idx],
                                    obsm=adata.obsm,
                                    uns=adata.uns)
        
        # Add the new AnnData object to the dictionary
        splitted_adatas[modality] = new_adata
    
    return splitted_adatas
```

Example 5 (python):
```python
path_to_auc_gene="/nfs/team205/heart/pyscenic/plus/AUCscore/foetal_aCM_n570_eRegulon_AUC_gene.csv"
path_to_auc_region="/nfs/team205/heart/pyscenic/plus/AUCscore/foetal_aCM_n570_eRegulon_AUC_region.csv"

paired_adata_with_time_path = "/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA/aCMs_WholeSample_paired_with_time.h5ad"

temporal_ordering='age_days_smoothed_once'
```

Example 6 (python):
```python
import scanpy as sc
import pandas as pd
```

Example 7 (python):
```python
adata=sc.read(paired_adata_with_time_path)
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
adata.X.data[:5]
```

Example 10 (unknown):
```unknown
array([2., 1., 1., 1., 1.], dtype=float32)
```

Example 11 (python):
```python
sc.pl.umap(adata,color=["seg","milestones","t",temporal_ordering,"fine_grain"],ncols=2)
```

Example 12 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 13 (python):
```python
sc.pl.umap(adata,color=["fine_grain"],ncols=1)
sc.pl.umap(adata,color=[temporal_ordering,"t"],ncols=1,title="sample age")

sc.pl.umap(adata,color=["fine_grain","age_days","t"],ncols=1, legend_loc=None)
```

Example 14 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 15 (unknown):
```unknown
WARNING: The title list is shorter than the number of panels. Using 'color' value instead for some plots.
```

Example 16 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 17 (python):
```python
adata.shape
```

Example 18 (unknown):
```unknown
(5664, 544641)
```

Example 19 (python):
```python
modality_column = "modality"
modality_values = ["Gene Expression", "Peaks"]
splitted_adatas = split_adata_by_modality(adata, modality_column, modality_values)

adata_gex = splitted_adatas["Gene Expression"]
print(adata_gex)

adata_atac = splitted_adatas["Peaks"]
print(adata_atac)
```

Example 20 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 36601
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
AnnData object with n_obs × n_vars = 5664 × 508040
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
```

Example 21 (python):
```python
adata_gex.X.data[:10]
```

Example 22 (unknown):
```unknown
array([2., 1., 1., 1., 1., 1., 8., 2., 1., 1.], dtype=float32)
```

Example 23 (python):
```python
adata_atac.X.data[:10]
```

Example 24 (unknown):
```unknown
array([2., 1., 1., 1., 1., 2., 2., 1., 1., 2.], dtype=float32)
```

Example 25 (python):
```python
adata_gex.layers["counts"]=adata_gex.X.copy()
adata_atac.layers["counts"]=adata_atac.X.copy()

sc.pp.normalize_total(adata_gex, target_sum=1e4)
sc.pp.normalize_total(adata_atac, target_sum=1e4)

sc.pp.log1p(adata_gex)
sc.pp.log1p(adata_atac)
```

Example 26 (python):
```python
WARNING: adata.X seems to be already log-transformed.
```

Example 27 (python):
```python
adata_gex.X.data[:10]
```

Example 28 (unknown):
```unknown
array([1.559231 , 1.0569503, 1.0569503, 1.0569503, 1.0569503, 1.0569503,
       2.7738788, 1.559231 , 1.0569503, 1.0569503], dtype=float32)
```

Example 29 (python):
```python
adata_atac.X.data[:10]
```

Example 30 (unknown):
```unknown
array([0.6963374 , 0.40759304, 0.40759304, 0.40759304, 0.40759304,
       0.6963374 , 0.6963374 , 0.40759304, 0.40759304, 0.6963374 ],
      dtype=float32)
```

Example 31 (python):
```python
# read in regulon scores
auc_gene = pd.read_csv(path_to_auc_gene, index_col=0)
print(auc_gene.shape)
auc_region = pd.read_csv(path_to_auc_region, index_col=0)
print(auc_region.shape)
```

Example 32 (unknown):
```unknown
(17302, 278)
(12964, 278)
```

Example 33 (python):
```python
# make regulon adata, AUC gene
# AUC Gene
# create anndata
adata_aucg = anndata.AnnData(
    X=auc_gene.loc[adata_gex.obs_names],
    obs=adata_gex.obs,
    var=pd.DataFrame(auc_gene.columns).set_index(0),
    obsm=adata_gex.obsm,
    uns= adata_gex.uns
)
# modify varnames
adata_aucg.var_names = [x.split('_(')[0] for x in adata_aucg.var_names]
print(adata_aucg.shape)

# AUC Region
# create anndata
adata_aucr = anndata.AnnData(
    X=auc_region.loc[adata_gex.obs_names],
    obs=adata_gex.obs,
    var=pd.DataFrame(auc_region.columns).set_index(0),
    obsm=adata_gex.obsm
)
# modify varnames
adata_aucr.var_names = [x.split('_(')[0] for x in adata_aucr.var_names]
print(adata_aucr.shape)
```

Example 34 (unknown):
```unknown
(5664, 278)
(5664, 278)
```

Example 35 (unknown):
```unknown
LEFT_or_RIGHT_features={}
```

Example 36 (unknown):
```unknown
log2fc_min=0.5
pval_cutoff=0.1
```

Example 37 (python):
```python
sc.tl.dendrogram(adata_gex,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_gex,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_gex,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
LEFT_or_RIGHT_features["LEFT_genes"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
LEFT_or_RIGHT_features["RIGHT_genes"]=df.names.tolist()
```

Example 38 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 39 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 40 (unknown):
```unknown
plot.GO_plot(gene_list=LEFT_or_RIGHT_features["RIGHT_genes"],display_table=True,
             gene_sets=['GO_Biological_Process_2021'],
             title_suffix=" AtrialCardiomyocytesRight",
            n_rank=10)
```

Example 41 (unknown):
```unknown
GO_Biological_Process_2021
```

Example 42 (unknown):
```unknown
plot.GO_plot(gene_list=LEFT_or_RIGHT_features["LEFT_genes"],display_table=True,gene_sets=['GO_Biological_Process_2023'],title_suffix=" AtrialCardiomyocytesLeft")
```

Example 43 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 44 (python):
```python
sc.tl.dendrogram(adata_atac,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_atac,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_atac,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_peaks"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_peaks"]=df.names.tolist()
```

Example 45 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 46 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 47 (unknown):
```unknown
adding 5316 features for LEFT...
adding 3871 features for RIGHT...
```

Example 48 (python):
```python
# Instead of doing scanpy DE testing, we use the results of ArchR for this step
df=pd.read_csv("./DAR_testing/L-vs-R_aCM.csv")
df=df[(df.Log2FC>log2fc_min)&(df.FDR<pval_cutoff)]
df["names"]=df["seqnames"].astype(str)+":"+df["start"].astype(str)+"_"+df["end"].astype(str)
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_peaks"]=df.names.tolist()

df=pd.read_csv("./DAR_testing/R-vs-L_aCM.csv")
df=df[(df.Log2FC>log2fc_min)&(df.FDR<pval_cutoff)]
df["names"]=df["seqnames"].astype(str)+":"+df["start"].astype(str)+"_"+df["end"].astype(str)
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_peaks"]=df.names.tolist()
```

Example 49 (unknown):
```unknown
adding 132 features for LEFT...
adding 57 features for RIGHT...
```

Example 50 (python):
```python
sc.tl.dendrogram(adata_aucg,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_aucg,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_aucg,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata_aucg,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_AUCgene"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata_aucg,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_AUCgene"]=df.names.tolist()
```

Example 51 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 52 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 53 (unknown):
```unknown
adding 11 features for LEFT...
adding 4 features for RIGHT...
```

Example 54 (python):
```python
sc.tl.dendrogram(adata_aucr,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_aucr,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_aucr,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata_aucr,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_AUCregion"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata_aucr,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_AUCregion"]=df.names.tolist()
```

Example 55 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 56 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 57 (unknown):
```unknown
adding 15 features for LEFT...
adding 0 features for RIGHT...
```

Example 58 (python):
```python
import scanpy as sc  # Import scanpy, assuming it's used for gene scoring and plotting

def plot_sided_features(adata, feature_type, associated_increasing_features_dict, cmap="RdPu", sides=["LEFT", "RIGHT", "COMMON","NON_SIDE_SPECIFIC"]):
    """
    Plots sided features for a given AnnData object and feature type.
    
    Args:
    adata (AnnData): An AnnData object containing gene expression data.
    feature_type (str): The type of feature to plot.
    associated_increasing_features_dict (dict): A dictionary containing associated increasing features.
    """
    # Copy the adata object to avoid modifying the original data
    plot_adata = adata.copy()

    # Initialize an empty list to store score names
    scores = []

    # Iterate over the sides and calculate scores for each
    for side in sides:
        # Retrieve the gene list from the dictionary
        gene_list = associated_increasing_features_dict[f"{feature_type}_{side}"]

        if gene_list:
            # Construct a unique name for the score
            score_name = f"{feature_type}_{side}_({len(gene_list)} features)"
            # Add the score name to the list
            scores.append(score_name)
            # Calculate the score using scanpy's score_genes function
            sc.tl.score_genes(plot_adata, gene_list=gene_list, score_name=score_name)

    # Plot the UMAP with the scores
    if len(scores)>0:
        sc.pl.umap(plot_adata, color=scores, ncols=3, wspace=0.8, cmap=cmap)
```

Example 59 (unknown):
```unknown
# New dictionary to store converted keys
LEFT_or_RIGHT_features_converted = {}

# Iterate through each item in LEFT_or_RIGHT_features
for key, value in LEFT_or_RIGHT_features.items():
    # Split the key on the underscore
    parts = key.split('_')
    
    # Rearrange the parts to match the desired format
    # New key format: 'genes_LEFT', 'genes_RIGHT', etc.
    new_key = f"{parts[1]}_{parts[0]}"

    # Add to the converted dictionary
    LEFT_or_RIGHT_features_converted[new_key] = value
LEFT_or_RIGHT_features_converted.keys()
```

Example 60 (unknown):
```unknown
dict_keys(['genes_LEFT', 'genes_RIGHT', 'peaks_LEFT', 'peaks_RIGHT', 'AUCgene_LEFT', 'AUCgene_RIGHT', 'AUCregion_LEFT', 'AUCregion_RIGHT'])
```

Example 61 (python):
```python
plot_sided_features(adata_gex,
                    sides=["LEFT","RIGHT"],
                    feature_type="genes",
                    associated_increasing_features_dict=LEFT_or_RIGHT_features_converted)

plot_sided_features(adata_atac,
                    sides=["LEFT","RIGHT"],
                    feature_type="peaks",
                    associated_increasing_features_dict=LEFT_or_RIGHT_features_converted)

plot_sided_features(adata_aucg,
                    sides=["LEFT","RIGHT"],
                    feature_type="AUCgene",
                    associated_increasing_features_dict=LEFT_or_RIGHT_features_converted)

plot_sided_features(adata_aucr,
                    sides=["LEFT","RIGHT"],
                    feature_type="AUCregion",
                    associated_increasing_features_dict=LEFT_or_RIGHT_features_converted)
```

Example 62 (python):
```python
print(LEFT_or_RIGHT_features_converted["AUCgene_LEFT"])
print(LEFT_or_RIGHT_features_converted["AUCregion_LEFT"])
print(LEFT_or_RIGHT_features_converted["AUCgene_RIGHT"])
```

Example 63 (unknown):
```unknown
['PITX2_+', 'ETS2_-', 'MAF_-', 'GATA5_+', 'SIX5_extended_+', 'ZNF337_+', 'PEG3_+', 'NFATC1_+', 'ZNF383_-', 'ARID3A_extended_-', 'MEF2A_-']
['GTF2IRD1_+', 'PITX2_+', 'IKZF2_+', 'NFE2L1_+', 'NFE2L2_+', 'TCF7L2_+', 'MGA_+', 'FOXJ2_+', 'MAF_-', 'STAT5B_+', 'MESP1_extended_+', 'ETS2_-', 'STAT4_+', 'ZNF337_+', 'ZNF470_+']
['ETS2_+', 'MAF_+', 'MYCN_+', 'ZNF526_+']
```

Example 64 (python):
```python
import pandas as pd
import numpy as np
from typing import Any

def bin_pseudotime(adata: Any, pseudotime_column: str, n_bins: int) -> Any:
    """
    Bin cells in an AnnData object into n_bins based on a specified pseudotime column.

    Parameters:
    adata (AnnData): The AnnData object containing single-cell data.
    pseudotime_column (str): The name of the column in adata.obs that contains pseudotime values.
    n_bins (int): The number of bins to divide the pseudotime values into.

    Returns:
    AnnData: The AnnData object with an additional column in adata.obs indicating the bin each cell belongs to.
    """
    # Calculate the percentiles that will be used as bin edges
    percentiles = [i*(100/n_bins) for i in range(1, n_bins)]
    bin_edges = np.percentile(adata.obs[pseudotime_column], percentiles)
    
    # Bin values into discrete intervals using pandas.cut
    adata.obs[f'{pseudotime_column}_bins'] = pd.cut(
        adata.obs[pseudotime_column],
        bins=np.insert(bin_edges, [0, len(bin_edges)], [adata.obs[pseudotime_column].min(), adata.obs[pseudotime_column].max()]),  # Include the extreme values of the pseudotime range
        include_lowest=True,
        right=True,
        labels=[str(i) for i in range(n_bins)]
    )
    
    # Return the modified AnnData object
    return adata
```

Example 65 (unknown):
```unknown
n_bins=2 # number of temporal bins
```

Example 66 (python):
```python
adata_gex = bin_pseudotime(adata_gex, 't', n_bins)
adata_atac = bin_pseudotime(adata_atac, 't', n_bins)

adata_gex = bin_pseudotime(adata_gex, temporal_ordering, n_bins)
adata_atac = bin_pseudotime(adata_atac, temporal_ordering, n_bins)

adata_aucg = bin_pseudotime(adata_aucg, 't', n_bins)
adata_aucr = bin_pseudotime(adata_aucg, 't', n_bins)

adata_aucg = bin_pseudotime(adata_aucr, temporal_ordering, n_bins)
adata_aucr = bin_pseudotime(adata_aucr, temporal_ordering, n_bins)

sc.pl.umap(adata_gex,color=["seg","milestones","t",temporal_ordering,"fine_grain","t_bins",f"{temporal_ordering}_bins"],ncols=2,wspace=0.8)
```

Example 67 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 68 (unknown):
```unknown
max_group=n_bins-1
max_group=str(max_group)
max_group
```

Example 69 (python):
```python
adata_gex.obs[f"{temporal_ordering}_bins"].value_counts()
```

Example 70 (unknown):
```unknown
0    2832
1    2832
Name: age_days_smoothed_once_bins, dtype: int64
```

Example 71 (python):
```python
increasing_features_dict={}

sc.tl.rank_genes_groups(adata_gex,groupby=f"{temporal_ordering}_bins",groups=[max_group],reference="0")
sc.pl.rank_genes_groups_dotplot(adata_gex,groupby=f"{temporal_ordering}_bins",n_genes=40,dendrogram=False)

df=sc.get.rank_genes_groups_df(adata_gex,group=max_group,log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
increasing_features_dict["INCREASING_genes"]=df.names.tolist()




sc.tl.rank_genes_groups(adata_atac,groupby=f"{temporal_ordering}_bins",groups=[max_group],reference="0")
sc.pl.rank_genes_groups_dotplot(adata_atac,groupby=f"{temporal_ordering}_bins",n_genes=40,dendrogram=False)

df=sc.get.rank_genes_groups_df(adata_atac,group=max_group,log2fc_min=log2fc_min)
#df=df[df.pvals_adj<pval_cutoff]
df=df[df.pvals<pval_cutoff] #pvals
increasing_features_dict["INCREASING_peaks"]=df.names.tolist()




sc.tl.rank_genes_groups(adata_aucg,groupby=f"{temporal_ordering}_bins",groups=[max_group],reference="0")
sc.pl.rank_genes_groups_dotplot(adata_aucg,groupby=f"{temporal_ordering}_bins",n_genes=40,dendrogram=False)

df=sc.get.rank_genes_groups_df(adata_aucg,group=max_group,log2fc_min=log2fc_min)
#df=df[df.pvals_adj<pval_cutoff]
df=df[df.pvals<pval_cutoff] #pvals
increasing_features_dict["INCREASING_AUCgene"]=df.names.tolist()




sc.tl.rank_genes_groups(adata_aucr,groupby=f"{temporal_ordering}_bins",groups=[max_group],reference="0")
sc.pl.rank_genes_groups_dotplot(adata_aucr,groupby=f"{temporal_ordering}_bins",n_genes=40,dendrogram=False)

df=sc.get.rank_genes_groups_df(adata_aucr,group=max_group,log2fc_min=log2fc_min)
#df=df[df.pvals_adj<pval_cutoff]
df=df[df.pvals<pval_cutoff] #pvals
increasing_features_dict["INCREASING_AUCregion"]=df.names.tolist()
```

Example 72 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
```

Example 73 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 74 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
```

Example 75 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 76 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
```

Example 77 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 78 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
```

Example 79 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 80 (unknown):
```unknown
plot.GO_plot(gene_list=increasing_features_dict["INCREASING_genes"],display_table=True,
             gene_sets=['MSigDB_Hallmark_2020'],
             title_suffix=" AtrialCardiomyocytesCommon increasing over time")
```

Example 81 (unknown):
```unknown
MSigDB_Hallmark_2020
```

Example 82 (unknown):
```unknown
increasing_features_dict
```

Example 83 (python):
```python
from tqdm import tqdm
import pandas as pd

sort_col = "A"
cutoff_col = 'fdr'
cutoff_value = 1e-1

associated_increasing_features_dict = {}

for feature in tqdm(["genes", "peaks", "AUCgene", "AUCregion"]):
    print("\n######### {} #########\n".format(feature))
    
    associated_increasing_NONsidespecific_features = []

    for side in ["LEFT", "RIGHT"]:
        # Load and sort the data
        df = pd.read_csv(f"./fitted_features/aCMs_WholeSample_scfatespseudotime_{feature}_{side}.csv", index_col=0)
        df = df.sort_values(sort_col, ascending=False).drop(columns="activator_repressor")
        print(f"Total number of {feature}_{side} before filtering: {len(df)}")
        
        # Apply cutoff and filter positive 'A' values
        associated_features = df[(df[cutoff_col] < cutoff_value) & (df["A"] > 0.5)].index.tolist()
        print(f"Number of {feature}_{side} after cutoff and 'A' filtering: {len(associated_features)}")
        
        # Filter for increasing features
        increasing_features = increasing_features_dict[f"INCREASING_{feature}"]
        print(f"Number of increasing {feature}: {len(increasing_features)}")
        
        # Find associated increasing features
        associated_increasing_features = [f for f in associated_features if f in increasing_features]
        print(f"Number of increasing {feature}_{side} after associating: {len(associated_increasing_features)}")
        
        # Filter for side-specific features
        side_specific_features = LEFT_or_RIGHT_features[f"{side}_{feature}"]
        print(f"Number of {side}-specific {feature}: {len(side_specific_features)}")
        
        # Find associated increasing side-specific features
        associated_increasing_sidespecific_features = [f for f in associated_increasing_features if f in side_specific_features]
        print(f"Number of associated increasing {side}-specific {feature}: {len(associated_increasing_sidespecific_features)}")
        
        # Find non-side-specific features
        non_side_specific_features = [f for f in associated_increasing_features if f not in side_specific_features]
        print(f"Number of associated increasing non-{side}-specific {feature}: {len(non_side_specific_features)}")
        
        # Combine non-side-specific features for later processing
        associated_increasing_NONsidespecific_features.extend(non_side_specific_features)
        
        # Store side-specific features
        associated_increasing_features_dict[f"{feature}_{side}"] = associated_increasing_sidespecific_features

    # Deduplicate the list of non-side-specific features
    non_side_specific_set = set(associated_increasing_NONsidespecific_features)
    print(f"Total number of associated increasing non-side-specific {feature}: {len(non_side_specific_set)}")

    # Store these NonSideSpecific features
    associated_increasing_features_dict[f"{feature}_NON_SIDE_SPECIFIC"] = list(non_side_specific_set)

    # Identify common features (present in both LEFT and RIGHT)
    common_features = non_side_specific_set.intersection(
        associated_increasing_features_dict[f"{feature}_LEFT"],
        associated_increasing_features_dict[f"{feature}_RIGHT"]
    )
    associated_increasing_features_dict[f"{feature}_COMMON"] = list(common_features)
    print(f"Total number of common non-side-specific {feature}: {len(common_features)}")

# Calculate and display counts for each category
counts_dict = {key: len(value) for key, value in associated_increasing_features_dict.items()}
print("\nCounts per category:")
for key, value in counts_dict.items():
    print(f"{key}: {value}")
```

Example 84 (unknown):
```unknown
0%|                                                                                                                                                                                                                     | 0/4 [00:00<?, ?it/s]
```

Example 85 (unknown):
```unknown
######### genes #########

Total number of genes_LEFT before filtering: 21922
Number of genes_LEFT after cutoff and 'A' filtering: 3626
Number of increasing genes: 1178
Number of increasing genes_LEFT after associating: 577
Number of LEFT-specific genes: 1440
```

Example 86 (unknown):
```unknown
25%|███████████████████████████████████████████████████▎                                                                                                                                                         | 1/4 [00:00<00:01,  2.77it/s]
```

Example 87 (unknown):
```unknown
Number of associated increasing LEFT-specific genes: 98
Number of associated increasing non-LEFT-specific genes: 479
Total number of genes_RIGHT before filtering: 21024
Number of genes_RIGHT after cutoff and 'A' filtering: 3439
Number of increasing genes: 1178
Number of increasing genes_RIGHT after associating: 485
Number of RIGHT-specific genes: 936
Number of associated increasing RIGHT-specific genes: 83
Number of associated increasing non-RIGHT-specific genes: 402
Total number of associated increasing non-side-specific genes: 569
Total number of common non-side-specific genes: 0

######### peaks #########

Total number of peaks_LEFT before filtering: 167061
Number of peaks_LEFT after cutoff and 'A' filtering: 4835
Number of increasing peaks: 5856
Number of increasing peaks_LEFT after associating: 113
Number of LEFT-specific peaks: 132
Number of associated increasing LEFT-specific peaks: 1
Number of associated increasing non-LEFT-specific peaks: 112
Total number of peaks_RIGHT before filtering: 166063
Number of peaks_RIGHT after cutoff and 'A' filtering: 7171
Number of increasing peaks: 5856
```

Example 88 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:02<00:00,  1.90it/s]
```

Example 89 (unknown):
```unknown
Number of increasing peaks_RIGHT after associating: 166
Number of RIGHT-specific peaks: 57
Number of associated increasing RIGHT-specific peaks: 0
Number of associated increasing non-RIGHT-specific peaks: 166
Total number of associated increasing non-side-specific peaks: 262
Total number of common non-side-specific peaks: 0

######### AUCgene #########

Total number of AUCgene_LEFT before filtering: 278
Number of AUCgene_LEFT after cutoff and 'A' filtering: 83
Number of increasing AUCgene: 4
Number of increasing AUCgene_LEFT after associating: 4
Number of LEFT-specific AUCgene: 11
Number of associated increasing LEFT-specific AUCgene: 0
Number of associated increasing non-LEFT-specific AUCgene: 4
Total number of AUCgene_RIGHT before filtering: 278
Number of AUCgene_RIGHT after cutoff and 'A' filtering: 85
Number of increasing AUCgene: 4
Number of increasing AUCgene_RIGHT after associating: 4
Number of RIGHT-specific AUCgene: 4
Number of associated increasing RIGHT-specific AUCgene: 0
Number of associated increasing non-RIGHT-specific AUCgene: 4
Total number of associated increasing non-side-specific AUCgene: 4
Total number of common non-side-specific AUCgene: 0

######### AUCregion #########

Total number of AUCregion_LEFT before filtering: 278
Number of AUCregion_LEFT after cutoff and 'A' filtering: 75
Number of increasing AUCregion: 4
Number of increasing AUCregion_LEFT after associating: 3
Number of LEFT-specific AUCregion: 15
Number of associated increasing LEFT-specific AUCregion: 0
Number of associated increasing non-LEFT-specific AUCregion: 3
Total number of AUCregion_RIGHT before filtering: 278
Number of AUCregion_RIGHT after cutoff and 'A' filtering: 72
Number of increasing AUCregion: 4
Number of increasing AUCregion_RIGHT after associating: 3
Number of RIGHT-specific AUCregion: 0
Number of associated increasing RIGHT-specific AUCregion: 0
Number of associated increasing non-RIGHT-specific AUCregion: 3
Total number of associated increasing non-side-specific AUCregion: 4
Total number of common non-side-specific AUCregion: 0

Counts per category:
genes_LEFT: 98
genes_RIGHT: 83
genes_NON_SIDE_SPECIFIC: 569
genes_COMMON: 0
peaks_LEFT: 1
peaks_RIGHT: 0
peaks_NON_SIDE_SPECIFIC: 262
peaks_COMMON: 0
AUCgene_LEFT: 0
AUCgene_RIGHT: 0
AUCgene_NON_SIDE_SPECIFIC: 4
AUCgene_COMMON: 0
AUCregion_LEFT: 0
AUCregion_RIGHT: 0
AUCregion_NON_SIDE_SPECIFIC: 4
AUCregion_COMMON: 0
```

Example 90 (python):
```python
plot_sided_features(adata_gex,
                    feature_type="genes",
                    associated_increasing_features_dict=associated_increasing_features_dict)

plot_sided_features(adata_atac,
                    feature_type="peaks",
                    associated_increasing_features_dict=associated_increasing_features_dict)

plot_sided_features(adata_aucg,
                    feature_type="AUCgene",
                    associated_increasing_features_dict=associated_increasing_features_dict)

plot_sided_features(adata_aucr,
                    feature_type="AUCregion",
                    associated_increasing_features_dict=associated_increasing_features_dict)
```

Example 91 (unknown):
```unknown
LEFT_or_RIGHT_features={}
```

Example 92 (unknown):
```unknown
log2fc_min=2
pval_cutoff=0.05
```

Example 93 (python):
```python
sc.tl.dendrogram(adata_gex,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_gex,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_gex,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
LEFT_or_RIGHT_features["LEFT_genes"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
LEFT_or_RIGHT_features["RIGHT_genes"]=df.names.tolist()
```

Example 94 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 95 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 96 (python):
```python
sc.tl.dendrogram(adata_atac,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_atac,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_atac,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_peaks"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_peaks"]=df.names.tolist()
```

Example 97 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 98 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 99 (unknown):
```unknown
adding 1100 features for LEFT...
adding 414 features for RIGHT...
```

Example 100 (python):
```python
# Instead of doing scanpy DE testing, we use the results of ArchR for this step
df=pd.read_csv("./DAR_testing/L-vs-R_aCM.csv")
df=df[(df.Log2FC>log2fc_min)&(df.FDR<pval_cutoff)]
df["names"]=df["seqnames"].astype(str)+":"+df["start"].astype(str)+"_"+df["end"].astype(str)
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_peaks"]=df.names.tolist()

df=pd.read_csv("./DAR_testing/R-vs-L_aCM.csv")
df=df[(df.Log2FC>log2fc_min)&(df.FDR<pval_cutoff)]
df["names"]=df["seqnames"].astype(str)+":"+df["start"].astype(str)+"_"+df["end"].astype(str)
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_peaks"]=df.names.tolist()
```

Example 101 (unknown):
```unknown
adding 71 features for LEFT...
adding 19 features for RIGHT...
```

Example 102 (python):
```python
sc.tl.dendrogram(adata_aucg,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_aucg,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_aucg,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata_aucg,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_AUCgene"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata_aucg,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_AUCgene"]=df.names.tolist()
```

Example 103 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 104 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 105 (unknown):
```unknown
adding 0 features for LEFT...
adding 0 features for RIGHT...
```

Example 106 (python):
```python
sc.tl.dendrogram(adata_aucr,groupby="fine_grain",use_rep="X_multiVI")
sc.tl.rank_genes_groups(adata_aucr,groupby="fine_grain")
sc.pl.rank_genes_groups_dotplot(adata_aucr,groupby="fine_grain",n_genes=20)

df=sc.get.rank_genes_groups_df(adata_aucr,group="AtrialCardiomyocytesLeft",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for LEFT...")
LEFT_or_RIGHT_features["LEFT_AUCregion"]=df.names.tolist()

df=sc.get.rank_genes_groups_df(adata_aucr,group="AtrialCardiomyocytesRight",log2fc_min=log2fc_min)
df=df[df.pvals_adj<pval_cutoff]
print(f"adding {len(df.names.tolist())} features for RIGHT...")
LEFT_or_RIGHT_features["RIGHT_AUCregion"]=df.names.tolist()
```

Example 107 (python):
```python
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'
WARNING: Dendrogram not added. Dendrogram is added only when the number of categories to plot > 2
```

Example 108 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

Example 109 (unknown):
```unknown
adding 0 features for LEFT...
adding 0 features for RIGHT...
```

Example 110 (python):
```python
from tqdm import tqdm
import pandas as pd

sort_col="A"
cutff_col = 'fdr'
cutoff_value = 1e-1

associated_features_dict = {}

for feature in tqdm(["genes", "peaks", "AUCgene", "AUCregion"]):
    print("\n")
    print(f"######### {feature} #########")
    print("\n")
    
    associated_increasing_NONsidespecific_features = []

    for side in ["LEFT", "RIGHT"]:
        # Read and sort the CSV file
        df = pd.read_csv(f"./fitted_features/aCMs_WholeSample_scfatespseudotime_{feature}_{side}.csv", index_col=0).sort_values(sort_col,ascending=False)
        df = df.drop(columns="activator_repressor")
        print(f"Total number of {feature}_{side} before filtering: {len(df)}")
        
        associated_features = df[(df[cutff_col] < cutoff_value) & (df["A"] > 1)].index.tolist()
        print(f"Number of {feature}_{side} after cutoff and 'A' filtering: {len(associated_features)}")
        
        # Filter for side-specific features
        side_specific_features = LEFT_or_RIGHT_features[f"{side}_{feature}"]
        print(f"Number of {side}-specific {feature}: {len(side_specific_features)}")
        
        associated_sidespecific_features = [f for f in associated_features if f in side_specific_features]
        print(f"Number of associated {side}-specific {feature}: {len(associated_sidespecific_features)}")
        
        associated_features_dict[f"{feature}_{side}"] = associated_sidespecific_features

    # Read in COMMON features which are associated with pseudotime
    df = pd.read_csv(f"./fitted_features/aCMs_WholeSample_scfatespseudotime_{feature}_COMMON.csv", index_col=0).sort_values(sort_col,ascending=False)
    df = df.drop(columns="activator_repressor")
    print(f"Total number of {feature}_COMMON before filtering: {len(df)}")
    
    associated_features = df[(df[cutff_col] < cutoff_value) & (df["A"] > 1)].index.tolist()
    print(f"Number of {feature}_COMMON after cutoff and 'A' filtering: {len(associated_features)}")

    # Filter out any features which are known to be side-specific
    LEFT_side_specific_features = LEFT_or_RIGHT_features[f"LEFT_{feature}"]
    RIGHT_side_specific_features = LEFT_or_RIGHT_features[f"RIGHT_{feature}"]
    LEFT_or_RIGHT_side_specific_features=LEFT_side_specific_features+RIGHT_side_specific_features
    print(f"Number of (either) side-specific {feature}: {len(LEFT_or_RIGHT_side_specific_features)}")

    associated_non_sidespecific_features = [f for f in associated_features if f not in LEFT_or_RIGHT_side_specific_features]
    print(f"Number of associated non-side-specific {feature}: {len(associated_non_sidespecific_features)}")

    associated_features_dict[f"{feature}_COMMON"]=associated_non_sidespecific_features

# Calculate the counts for each category
counts_dict = {key: len(value) for key, value in associated_features_dict.items()}
print("\nCounts per category:")
for key, value in counts_dict.items():
    print(f"{key}: {value}")
```

Example 111 (unknown):
```unknown
0%|                                                                                                                                                                                                                     | 0/4 [00:00<?, ?it/s]
```

Example 112 (unknown):
```unknown
######### genes #########


Total number of genes_LEFT before filtering: 21922
Number of genes_LEFT after cutoff and 'A' filtering: 2519
Number of LEFT-specific genes: 209
Number of associated LEFT-specific genes: 28
Total number of genes_RIGHT before filtering: 21024
Number of genes_RIGHT after cutoff and 'A' filtering: 2584
Number of RIGHT-specific genes: 129
Number of associated RIGHT-specific genes: 29
Total number of genes_COMMON before filtering: 23953
```

Example 113 (unknown):
```unknown
25%|███████████████████████████████████████████████████▎                                                                                                                                                         | 1/4 [00:00<00:00,  5.31it/s]
```

Example 114 (unknown):
```unknown
Number of genes_COMMON after cutoff and 'A' filtering: 1081
Number of (either) side-specific genes: 338
Number of associated non-side-specific genes: 1048


######### peaks #########


Total number of peaks_LEFT before filtering: 167061
Number of peaks_LEFT after cutoff and 'A' filtering: 1122
Number of LEFT-specific peaks: 71
Number of associated LEFT-specific peaks: 0
Total number of peaks_RIGHT before filtering: 166063
Number of peaks_RIGHT after cutoff and 'A' filtering: 3933
Number of RIGHT-specific peaks: 19
Number of associated RIGHT-specific peaks: 0
```

Example 115 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00,  3.84it/s]
```

Example 116 (unknown):
```unknown
Total number of peaks_COMMON before filtering: 165795
Number of peaks_COMMON after cutoff and 'A' filtering: 0
Number of (either) side-specific peaks: 90
Number of associated non-side-specific peaks: 0


######### AUCgene #########


Total number of AUCgene_LEFT before filtering: 278
Number of AUCgene_LEFT after cutoff and 'A' filtering: 62
Number of LEFT-specific AUCgene: 0
Number of associated LEFT-specific AUCgene: 0
Total number of AUCgene_RIGHT before filtering: 278
Number of AUCgene_RIGHT after cutoff and 'A' filtering: 69
Number of RIGHT-specific AUCgene: 0
Number of associated RIGHT-specific AUCgene: 0
Total number of AUCgene_COMMON before filtering: 278
Number of AUCgene_COMMON after cutoff and 'A' filtering: 52
Number of (either) side-specific AUCgene: 0
Number of associated non-side-specific AUCgene: 52


######### AUCregion #########


Total number of AUCregion_LEFT before filtering: 278
Number of AUCregion_LEFT after cutoff and 'A' filtering: 50
Number of LEFT-specific AUCregion: 0
Number of associated LEFT-specific AUCregion: 0
Total number of AUCregion_RIGHT before filtering: 278
Number of AUCregion_RIGHT after cutoff and 'A' filtering: 58
Number of RIGHT-specific AUCregion: 0
Number of associated RIGHT-specific AUCregion: 0
Total number of AUCregion_COMMON before filtering: 278
Number of AUCregion_COMMON after cutoff and 'A' filtering: 56
Number of (either) side-specific AUCregion: 0
Number of associated non-side-specific AUCregion: 56

Counts per category:
genes_LEFT: 28
genes_RIGHT: 29
genes_COMMON: 1048
peaks_LEFT: 0
peaks_RIGHT: 0
peaks_COMMON: 0
AUCgene_LEFT: 0
AUCgene_RIGHT: 0
AUCgene_COMMON: 52
AUCregion_LEFT: 0
AUCregion_RIGHT: 0
AUCregion_COMMON: 56
```

Example 117 (python):
```python
# similar to before but we do not filter using the increasing features dictionary
from tqdm import tqdm
import pandas as pd

sort_col = "A"
cutoff_col = 'fdr'
cutoff_value = 1e-1

associated_increasing_features_dict = {}

for feature in tqdm(["genes", "peaks", "AUCgene", "AUCregion"]):
    print("\n######### {} #########\n".format(feature))
    
    associated_increasing_NONsidespecific_features = []

    for side in ["LEFT", "RIGHT"]:
        # Load and sort the data
        df = pd.read_csv(f"./fitted_features/aCMs_WholeSample_scfatespseudotime_{feature}_{side}.csv", index_col=0)
        df = df.sort_values(sort_col, ascending=False).drop(columns="activator_repressor")
        print(f"Total number of {feature}_{side} before filtering: {len(df)}")
        
        # Apply cutoff and filter positive 'A' values
        associated_features = df[(df[cutoff_col] < cutoff_value) & (df["A"] > 0.5)].index.tolist()
        print(f"Number of {feature}_{side} after cutoff and 'A' filtering: {len(associated_features)}")
        
        # Filter for increasing features
        #increasing_features = increasing_features_dict[f"INCREASING_{feature}"]
        #print(f"Number of increasing {feature}: {len(increasing_features)}")
        
        # Find associated increasing features
        #associated_increasing_features = [f for f in associated_features if f in increasing_features]
        #print(f"Number of increasing {feature}_{side} after associating: {len(associated_increasing_features)}")

        associated_increasing_features=associated_features
        
        # Filter for side-specific features
        side_specific_features = LEFT_or_RIGHT_features[f"{side}_{feature}"]
        print(f"Number of {side}-specific {feature}: {len(side_specific_features)}")
        
        # Find associated increasing side-specific features
        associated_increasing_sidespecific_features = [f for f in associated_increasing_features if f in side_specific_features]
        print(f"Number of associated increasing {side}-specific {feature}: {len(associated_increasing_sidespecific_features)}")
        
        # Find non-side-specific features
        non_side_specific_features = [f for f in associated_increasing_features if f not in side_specific_features]
        print(f"Number of associated increasing non-{side}-specific {feature}: {len(non_side_specific_features)}")
        
        # Combine non-side-specific features for later processing
        associated_increasing_NONsidespecific_features.extend(non_side_specific_features)
        
        # Store side-specific features
        associated_increasing_features_dict[f"{feature}_{side}"] = associated_increasing_sidespecific_features

    # Deduplicate the list of non-side-specific features
    non_side_specific_set = set(associated_increasing_NONsidespecific_features)
    print(f"Total number of associated increasing non-side-specific {feature}: {len(non_side_specific_set)}")

    # Store these NonSideSpecific features
    associated_increasing_features_dict[f"{feature}_NON_SIDE_SPECIFIC"] = list(non_side_specific_set)

    # Identify common features (present in both LEFT and RIGHT)
    common_features = non_side_specific_set.intersection(
        associated_increasing_features_dict[f"{feature}_LEFT"],
        associated_increasing_features_dict[f"{feature}_RIGHT"]
    )
    associated_increasing_features_dict[f"{feature}_COMMON"] = list(common_features)
    print(f"Total number of common non-side-specific {feature}: {len(common_features)}")

# Calculate and display counts for each category
counts_dict = {key: len(value) for key, value in associated_increasing_features_dict.items()}
print("\nCounts per category:")
for key, value in counts_dict.items():
    print(f"{key}: {value}")
```

Example 118 (unknown):
```unknown
0%|                                                                                                                                                                                                                     | 0/4 [00:00<?, ?it/s]
```

Example 119 (unknown):
```unknown
######### genes #########

Total number of genes_LEFT before filtering: 21922
Number of genes_LEFT after cutoff and 'A' filtering: 3626
Number of LEFT-specific genes: 209
Number of associated increasing LEFT-specific genes: 50
Number of associated increasing non-LEFT-specific genes: 3576
Total number of genes_RIGHT before filtering: 21024
Number of genes_RIGHT after cutoff and 'A' filtering: 3439
Number of RIGHT-specific genes: 129
Number of associated increasing RIGHT-specific genes: 42
Number of associated increasing non-RIGHT-specific genes: 3397
Total number of associated increasing non-side-specific genes: 4858
Total number of common non-side-specific genes: 0

######### peaks #########

Total number of peaks_LEFT before filtering: 167061
Number of peaks_LEFT after cutoff and 'A' filtering: 4835
Number of LEFT-specific peaks: 71
Number of associated increasing LEFT-specific peaks: 0
Number of associated increasing non-LEFT-specific peaks: 4835
```

Example 120 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 11.19it/s]
```

Example 121 (unknown):
```unknown
Total number of peaks_RIGHT before filtering: 166063
Number of peaks_RIGHT after cutoff and 'A' filtering: 7171
Number of RIGHT-specific peaks: 19
Number of associated increasing RIGHT-specific peaks: 0
Number of associated increasing non-RIGHT-specific peaks: 7171
Total number of associated increasing non-side-specific peaks: 11469
Total number of common non-side-specific peaks: 0

######### AUCgene #########

Total number of AUCgene_LEFT before filtering: 278
Number of AUCgene_LEFT after cutoff and 'A' filtering: 83
Number of LEFT-specific AUCgene: 0
Number of associated increasing LEFT-specific AUCgene: 0
Number of associated increasing non-LEFT-specific AUCgene: 83
Total number of AUCgene_RIGHT before filtering: 278
Number of AUCgene_RIGHT after cutoff and 'A' filtering: 85
Number of RIGHT-specific AUCgene: 0
Number of associated increasing RIGHT-specific AUCgene: 0
Number of associated increasing non-RIGHT-specific AUCgene: 85
Total number of associated increasing non-side-specific AUCgene: 104
Total number of common non-side-specific AUCgene: 0

######### AUCregion #########

Total number of AUCregion_LEFT before filtering: 278
Number of AUCregion_LEFT after cutoff and 'A' filtering: 75
Number of LEFT-specific AUCregion: 0
Number of associated increasing LEFT-specific AUCregion: 0
Number of associated increasing non-LEFT-specific AUCregion: 75
Total number of AUCregion_RIGHT before filtering: 278
Number of AUCregion_RIGHT after cutoff and 'A' filtering: 72
Number of RIGHT-specific AUCregion: 0
Number of associated increasing RIGHT-specific AUCregion: 0
Number of associated increasing non-RIGHT-specific AUCregion: 72
Total number of associated increasing non-side-specific AUCregion: 99
Total number of common non-side-specific AUCregion: 0

Counts per category:
genes_LEFT: 50
genes_RIGHT: 42
genes_NON_SIDE_SPECIFIC: 4858
genes_COMMON: 0
peaks_LEFT: 0
peaks_RIGHT: 0
peaks_NON_SIDE_SPECIFIC: 11469
peaks_COMMON: 0
AUCgene_LEFT: 0
AUCgene_RIGHT: 0
AUCgene_NON_SIDE_SPECIFIC: 104
AUCgene_COMMON: 0
AUCregion_LEFT: 0
AUCregion_RIGHT: 0
AUCregion_NON_SIDE_SPECIFIC: 99
AUCregion_COMMON: 0
```

Example 122 (python):
```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
from matplotlib import pyplot as plt
from matplotlib import cm
import numpy as np


def plot_fitted_expression(adata, gene_list, title, figsize, fname, show_colorbar=True):
    """
    Plots fitted gene expression over pseudotime for a list of genes.

    Parameters:
    adata (AnnData): Annotated data matrix of shape n_obs × n_vars. Rows correspond to cells and columns to genes.
    gene_list (list): List of genes to be plotted.
    title (str): Title of the plot.
    figsize (tuple): Figure size in the format (width, height).
    show_colorbar (bool): If True, display the color bar. Defaults to True.

    The function plots each gene's fitted expression as a function of pseudotime.
    The genes are ordered based on the pseudotime of their maximal expression.
    """
    pseudotime = adata.obs["t"].values  # Convert to numpy array
    pseudotime_reshaped = pseudotime.reshape(-1, 1)
    pseudotime_plot = np.linspace(min(pseudotime), max(pseudotime), 500)
    pseudotime_plot_reshaped = pseudotime_plot.reshape(-1, 1)
    gene_info = {}

    for gene in tqdm(gene_list, desc="Fitting expression..."):
        gene_expression = adata[:, gene].X.toarray()  # Adjust indexing based on your Anndata structure
        degrees = [1, 2, 3, 4, 5]
        best_score = -np.inf
        best_degree = 0

        for degree in degrees:
            model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
            score = cross_val_score(model, pseudotime_reshaped, gene_expression, cv=5).mean()
            if score > best_score:
                best_score = score
                best_degree = degree

        best_model = make_pipeline(PolynomialFeatures(best_degree), LinearRegression())
        best_model.fit(pseudotime_reshaped, gene_expression)
        fitted_expression_plot = best_model.predict(pseudotime_plot_reshaped)
        max_expression_idx = np.argmax(fitted_expression_plot)
        max_expression_time = pseudotime_plot[max_expression_idx]
        gene_info[gene] = {'model': best_model, 'max_time': max_expression_time}

    sorted_genes = sorted(gene_list, key=lambda gene: gene_info[gene]['max_time'], reverse=True)

    fig, ax = plt.subplots(figsize=figsize)
    cmap = cm.RdPu

    for i, gene in enumerate(sorted_genes):
        best_model = gene_info[gene]['model']
        fitted_expression_plot = best_model.predict(pseudotime_plot_reshaped)
        norm = plt.Normalize(fitted_expression_plot.min(), fitted_expression_plot.max())
        sc = plt.scatter(pseudotime_plot, np.ones_like(pseudotime_plot) * (i + 1), 
                         c=fitted_expression_plot, cmap=cmap, norm=norm, marker='s')

    if show_colorbar:
        plt.colorbar(sc, label='Fitted Gene Expression')

    # Remove the box around the plot
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['left'].set_visible(False)

    # Hide x-axis ticks and labels
    ax.set_xticks([])  # This removes the x-axis ticks
    ax.set_xlabel('')  # This removes the x-axis label

    # Adjust x-axis position
    ax.set_ylim(0.5, len(sorted_genes) + 0.5)

    plt.xlabel('Pseudotime')
    plt.ylabel('Gene')
    plt.yticks(np.arange(1, len(sorted_genes) + 1), sorted_genes, fontsize='xx-small', fontstyle='italic')
    plt.title(title)
    plt.savefig(fname=fname, bbox_inches='tight')
    plt.show()
```

Example 123 (python):
```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
from matplotlib import pyplot as plt
from matplotlib import cm
import numpy as np


def plot_fitted_expression(adata, gene_list, title, figsize, fname, show_colorbar=True):
    """
    Plots fitted gene expression over pseudotime for a list of genes.

    Parameters:
    adata (AnnData): Annotated data matrix of shape n_obs × n_vars. Rows correspond to cells and columns to genes.
    gene_list (list): List of genes to be plotted.
    title (str): Title of the plot.
    figsize (tuple): Figure size in the format (width, height).
    show_colorbar (bool): If True, display the color bar. Defaults to True.

    The function plots each gene's fitted expression as a function of pseudotime.
    The genes are ordered based on the pseudotime of their maximal expression.
    """
    pseudotime = adata.obs["t"].values  # Convert to numpy array

    # Normalize pseudotime to [0, 1] range
    pseudotime_min = min(pseudotime)
    pseudotime_max = max(pseudotime)
    pseudotime_normalized = (pseudotime - pseudotime_min) / (pseudotime_max - pseudotime_min)

    # Reshape for model fitting
    pseudotime_reshaped = pseudotime_normalized.reshape(-1, 1)
    
    # Create equally spaced values in [0, 1] for plotting
    pseudotime_plot = np.linspace(0, 1, 500)
    pseudotime_plot_reshaped = pseudotime_plot.reshape(-1, 1)
    
    gene_info = {}

    for gene in tqdm(gene_list, desc="Fitting expression..."):
        gene_expression = adata[:, gene].X.toarray()  # Adjust indexing based on your Anndata structure
        degrees = [1, 2, 3, 4, 5]
        best_score = -np.inf
        best_degree = 0

        for degree in degrees:
            model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
            score = cross_val_score(model, pseudotime_reshaped, gene_expression, cv=5).mean()
            if score > best_score:
                best_score = score
                best_degree = degree

        best_model = make_pipeline(PolynomialFeatures(best_degree), LinearRegression())
        best_model.fit(pseudotime_reshaped, gene_expression)
        fitted_expression_plot = best_model.predict(pseudotime_plot_reshaped)
        max_expression_idx = np.argmax(fitted_expression_plot)
        max_expression_time = pseudotime_plot[max_expression_idx]
        gene_info[gene] = {'model': best_model, 'max_time': max_expression_time}

    sorted_genes = sorted(gene_list, key=lambda gene: gene_info[gene]['max_time'], reverse=True)

    fig, ax = plt.subplots(figsize=figsize)
    cmap = cm.RdPu

    for i, gene in enumerate(sorted_genes):
        best_model = gene_info[gene]['model']
        fitted_expression_plot = best_model.predict(pseudotime_plot_reshaped)
        norm = plt.Normalize(fitted_expression_plot.min(), fitted_expression_plot.max())
        sc = plt.scatter(pseudotime_plot, np.ones_like(pseudotime_plot) * (i + 1), 
                         c=fitted_expression_plot, cmap=cmap, norm=norm, marker='s')

    if show_colorbar:
        plt.colorbar(sc, label='Fitted Gene Expression')

    # Remove the box around the plot
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['left'].set_visible(False)

    # If you wish, hide x-axis ticks and labels by uncommenting
    #ax.set_xticks([])  # This removes the x-axis ticks
    #ax.set_xlabel('')  # This removes the x-axis label
    
    ax.set_xlim(0, 1)
    ax.set_ylim(0.5, len(sorted_genes) + 0.5)

    plt.xlabel('Pseudotime')
    plt.ylabel('Gene')
    plt.yticks(np.arange(1, len(sorted_genes) + 1), sorted_genes, fontsize='xx-small', fontstyle='italic')
    plt.title(title)
    plt.savefig(fname=fname, bbox_inches='tight')
    plt.show()
```

Example 124 (unknown):
```unknown
associated_increasing_features_dict.keys()
```

Example 125 (unknown):
```unknown
dict_keys(['genes_LEFT', 'genes_RIGHT', 'genes_NON_SIDE_SPECIFIC', 'genes_COMMON', 'peaks_LEFT', 'peaks_RIGHT', 'peaks_NON_SIDE_SPECIFIC', 'peaks_COMMON', 'AUCgene_LEFT', 'AUCgene_RIGHT', 'AUCgene_NON_SIDE_SPECIFIC', 'AUCgene_COMMON', 'AUCregion_LEFT', 'AUCregion_RIGHT', 'AUCregion_NON_SIDE_SPECIFIC', 'AUCregion_COMMON'])
```

Example 126 (python):
```python
figsize=(2,4.5)
top_n_features=28
file_extension=".png"
file_extension=".svg"

plot_fitted_expression(adata_gex[adata_gex.obs.fine_grain=="AtrialCardiomyocytesLeft"], gene_list=associated_features_dict["genes_LEFT"][:top_n_features], title="LEFT atrial cardiomyocyte maturation", figsize=figsize, show_colorbar=False,fname=f"genes_LEFT{file_extension}")
plot_fitted_expression(adata_gex[adata_gex.obs.fine_grain=="AtrialCardiomyocytesRight"], gene_list=associated_features_dict["genes_RIGHT"][:top_n_features], title="RIGHT atrial cardiomyocyte maturation", figsize=figsize, show_colorbar=False,fname=f"genes_RIGHT{file_extension}")

#top_n_features=80
#figsize=(2,10)

plot_fitted_expression(adata_gex, gene_list=associated_features_dict["genes_COMMON"][:top_n_features], title="COMMON atrial cardiomyocyte maturation", figsize=figsize, show_colorbar=False,fname=f"genes_COMMON{file_extension}")
```

Example 127 (unknown):
```unknown
Fitting expression...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:02<00:00, 10.86it/s]
```

Example 128 (unknown):
```unknown
Fitting expression...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:01<00:00, 18.62it/s]
```

Example 129 (unknown):
```unknown
Fitting expression...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:01<00:00, 15.92it/s]
```

Example 130 (python):
```python
gene_list=associated_features_dict["genes_LEFT"]
adata_subset=adata_gex[adata_gex.obs.fine_grain=="AtrialCardiomyocytesLeft"].copy()
distance_threshold=4.5
```

Example 131 (python):
```python
import seaborn as sns
import pandas as pd
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, MinMaxScaler
from sklearn.linear_model import LinearRegression
from scipy.spatial.distance import pdist
from scipy.cluster.hierarchy import linkage, fcluster
from tqdm import tqdm
import matplotlib.pyplot as plt

def create_clustermap(adata, gene_list, n_bins, figsize, desired_n_clusters):
    """
    Creates a clustermap of fitted gene expression across pseudotime bins for a list of genes.

    Parameters:
    adata (AnnData): Annotated data matrix.
    gene_list (list): List of genes to be included.
    n_bins (int): Number of bins to divide pseudotime into.
    figsize (tuple): Figure size in the format (width, height).
    """
    # Normalize and reshape pseudotime
    pseudotime = adata.obs["t"].values
    pseudotime_min = min(pseudotime)
    pseudotime_max = max(pseudotime)
    pseudotime_normalized = (pseudotime - pseudotime_min) / (pseudotime_max - pseudotime_min)
    pseudotime_reshaped = pseudotime_normalized.reshape(-1, 1)

    # Create pseudotime bins
    bins = np.linspace(0, 1, n_bins + 1)
    pseudotime_binned = np.digitize(pseudotime_normalized, bins) - 1

    # DataFrame to store fitted expressions
    expression_data = pd.DataFrame(index=gene_list, columns=range(n_bins))

    print(f"Creating clustermap for {len(gene_list)} features across {n_bins} bins")

    for gene in tqdm(gene_list, desc="Fitting expression..."):
        gene_expression = adata[:, gene].X.toarray()

        # Check if gene_expression is empty or almost empty
        if gene_expression.size == 0 or np.count_nonzero(gene_expression) < 5:
            print(f"Insufficient data for gene: {gene}")
            continue

        # Fit the model
        model = make_pipeline(PolynomialFeatures(2), LinearRegression())
        model.fit(pseudotime_reshaped, gene_expression)

        # Predict expression for each bin
        for i in range(n_bins):
            bin_range = np.linspace(bins[i], bins[i + 1], 10).reshape(-1, 1)
            predicted_expression = model.predict(bin_range).mean()
            expression_data.loc[gene, i] = predicted_expression  # Direct assignment
    
    expression_data = expression_data.astype(float).fillna(0)

    # Perform min-max normalization
    scaler = MinMaxScaler()
    expression_data_normalized = pd.DataFrame(scaler.fit_transform(expression_data.T).T, index=gene_list, columns=expression_data.columns) # row-wise (per gene) min-max normalisation


    # Compute the distance matrix for the rows (genes) and then the linkage matrix
    distance_matrix = pdist(expression_data_normalized)
    row_linkage = linkage(distance_matrix, method='average')

    # Define clusters using the row linkage
    clusters = fcluster(row_linkage, desired_n_clusters, criterion='maxclust')

    # Define clusters using a distance threshold
    clusters = fcluster(row_linkage, distance_threshold, criterion='distance')

    # Generate a color palette with seaborn and map cluster labels to colors
    cluster_colors = sns.color_palette("tab20", n_colors=len(np.unique(clusters)))
    row_colors = pd.Series(clusters, index=gene_list).map(dict(zip(np.unique(clusters), cluster_colors)))

    # Create clustermap with row colors
    g = sns.clustermap(expression_data_normalized, row_linkage=row_linkage, figsize=figsize, cmap='RdPu', yticklabels=True, xticklabels=True, col_cluster=False, row_colors=row_colors)
    g.cax.set_visible(False)
    
    plt.setp(g.ax_heatmap.get_yticklabels(), fontsize='xx-small', fontstyle='italic')
    plt.show()

    return row_linkage, clusters

row_linkage, clusters = create_clustermap(adata=adata_subset, gene_list=gene_list, n_bins=500, figsize=(4,30), desired_n_clusters=3)




from collections import defaultdict

def create_cluster_dict(clusters, gene_list):
    """
    Creates a dictionary mapping cluster names to lists of genes, adjusting for zero-based indexing.

    Parameters:
    clusters (array): Array of cluster assignments for each gene.
    gene_list (list): List of genes corresponding to the clusters.

    Returns:
    dict: Dictionary with cluster names as keys and lists of genes as values.
    """
    cluster_dict = defaultdict(list)
    for gene, cluster in zip(gene_list, clusters):
        #print(f"Gene: {gene}, Cluster: {cluster}")  # Diagnostic print
        cluster_name=f"cluster_{cluster - 1}"
        cluster_dict[cluster_name].append(gene)

    return dict(cluster_dict)

# Example usage
cluster_dict = create_cluster_dict(clusters, gene_list=gene_list)

# Check the clusters and the number of genes in each
for cluster, genes in cluster_dict.items():
    print(f"{cluster}: {len(genes)} genes")

cluster_dict.keys()
scores=[]
for cluster_name,cluster_genes in cluster_dict.items():
    score_name=f"{cluster_name}_({len(cluster_genes)} genes)"
    scores.append(score_name)
    sc.tl.score_genes(adata_gex,gene_list=cluster_genes,score_name=f"{score_name}")

sc.pl.umap(adata_gex, color=scores, ncols=4, wspace=0.8, cmap="RdPu")








import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

optimal_orders = {}

for score in scores:
    # Prepare the data
    tmp_df = adata_gex.obs[["t", score]]
    X = tmp_df["t"].values.reshape(-1, 1)
    y = tmp_df[score].values

    # Split the data into training and testing sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    best_order = 1
    best_mse = float('inf')

    # Test different polynomial orders
    for order in range(1, 6):
        model = make_pipeline(PolynomialFeatures(order), LinearRegression())
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        mse = mean_squared_error(y_test, predictions)

        if mse < best_mse:
            best_mse = mse
            best_order = order

    optimal_orders[score] = best_order






import matplotlib.pyplot as plt
import seaborn as sns

# Number of scores (subplots)
num_scores = len(scores)

# Create a figure with a row of subplots
fig, axes = plt.subplots(1, num_scores, figsize=(5 * num_scores, 5), sharey=True)

# Track the global min and max for the y-axis
global_min, global_max = float('inf'), float('-inf')

# Plot each score
for i, score in enumerate(scores):
    # Prepare the data
    tmp_df = adata_gex.obs[["t", score, "fine_grain"]]
    
    # Plot for each fine_grain category
    fine_grain_categories = tmp_df["fine_grain"].unique()
    for category in fine_grain_categories:
        category_data = tmp_df[tmp_df["fine_grain"] == category]
        sns.regplot(ax=axes[i], data=category_data, x="t", y=score, order=optimal_orders[score], scatter=False, label=category)
    
    # Set the title and add legend
    axes[i].set_title(score)
    axes[i].legend()

    # Update global min and max
    current_min, current_max = tmp_df[score].min(), tmp_df[score].max()
    global_min, global_max = min(global_min, current_min), max(global_max, current_max)

# Set the same y-axis limits for all subplots
for ax in axes:
    ax.set_ylim(global_min, global_max)

# Adjust layout
plt.tight_layout()
plt.show()
```

Example 132 (unknown):
```unknown
Creating clustermap for 28 features across 500 bins
```

Example 133 (unknown):
```unknown
Fitting expression...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:02<00:00, 10.55it/s]
```

Example 134 (unknown):
```unknown
cluster_3: 7 genes
cluster_2: 6 genes
cluster_4: 2 genes
cluster_6: 1 genes
cluster_0: 8 genes
cluster_5: 3 genes
cluster_1: 1 genes
```

Example 135 (unknown):
```unknown
cluster_dict["cluster_3"]
```

Example 136 (unknown):
```unknown
['MBP', 'KCNMB2', 'GAN', 'SLC8A3', 'A2M', 'KCNMB2-AS1', 'KSR1']
```

Example 137 (unknown):
```unknown
gene_set_A=cluster_dict["cluster_3"]+cluster_dict["cluster_4"]+cluster_dict["cluster_6"]
gene_set_A
```

Example 138 (unknown):
```unknown
['MBP',
 'KCNMB2',
 'GAN',
 'SLC8A3',
 'A2M',
 'KCNMB2-AS1',
 'KSR1',
 'PANCR',
 'PITX2',
 'DCBLD2']
```

Example 139 (python):
```python
gene_list=associated_features_dict["genes_RIGHT"]
adata_subset=adata_gex[adata_gex.obs.fine_grain=="AtrialCardiomyocytesRight"].copy()
distance_threshold=4.5
```

Example 140 (python):
```python
import seaborn as sns
import pandas as pd
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, MinMaxScaler
from sklearn.linear_model import LinearRegression
from scipy.spatial.distance import pdist
from scipy.cluster.hierarchy import linkage, fcluster
from tqdm import tqdm
import matplotlib.pyplot as plt

def create_clustermap(adata, gene_list, n_bins, figsize, desired_n_clusters):
    """
    Creates a clustermap of fitted gene expression across pseudotime bins for a list of genes.

    Parameters:
    adata (AnnData): Annotated data matrix.
    gene_list (list): List of genes to be included.
    n_bins (int): Number of bins to divide pseudotime into.
    figsize (tuple): Figure size in the format (width, height).
    """
    # Normalize and reshape pseudotime
    pseudotime = adata.obs["t"].values
    pseudotime_min = min(pseudotime)
    pseudotime_max = max(pseudotime)
    pseudotime_normalized = (pseudotime - pseudotime_min) / (pseudotime_max - pseudotime_min)
    pseudotime_reshaped = pseudotime_normalized.reshape(-1, 1)

    # Create pseudotime bins
    bins = np.linspace(0, 1, n_bins + 1)
    pseudotime_binned = np.digitize(pseudotime_normalized, bins) - 1

    # DataFrame to store fitted expressions
    expression_data = pd.DataFrame(index=gene_list, columns=range(n_bins))

    print(f"Creating clustermap for {len(gene_list)} features across {n_bins} bins")

    for gene in tqdm(gene_list, desc="Fitting expression..."):
        gene_expression = adata[:, gene].X.toarray()

        # Check if gene_expression is empty or almost empty
        if gene_expression.size == 0 or np.count_nonzero(gene_expression) < 5:
            print(f"Insufficient data for gene: {gene}")
            continue

        # Fit the model
        model = make_pipeline(PolynomialFeatures(2), LinearRegression())
        model.fit(pseudotime_reshaped, gene_expression)

        # Predict expression for each bin
        for i in range(n_bins):
            bin_range = np.linspace(bins[i], bins[i + 1], 10).reshape(-1, 1)
            predicted_expression = model.predict(bin_range).mean()
            expression_data.loc[gene, i] = predicted_expression  # Direct assignment
    
    expression_data = expression_data.astype(float).fillna(0)

    # Perform min-max normalization
    scaler = MinMaxScaler()
    expression_data_normalized = pd.DataFrame(scaler.fit_transform(expression_data.T).T, index=gene_list, columns=expression_data.columns) # row-wise (per gene) min-max normalisation


    # Compute the distance matrix for the rows (genes) and then the linkage matrix
    distance_matrix = pdist(expression_data_normalized)
    row_linkage = linkage(distance_matrix, method='average')

    # Define clusters using the row linkage
    clusters = fcluster(row_linkage, desired_n_clusters, criterion='maxclust')

    # Define clusters using a distance threshold
    clusters = fcluster(row_linkage, distance_threshold, criterion='distance')

    # Generate a color palette with seaborn and map cluster labels to colors
    cluster_colors = sns.color_palette("tab20", n_colors=len(np.unique(clusters)))
    row_colors = pd.Series(clusters, index=gene_list).map(dict(zip(np.unique(clusters), cluster_colors)))

    # Create clustermap with row colors
    g = sns.clustermap(expression_data_normalized, row_linkage=row_linkage, figsize=figsize, cmap='RdPu', yticklabels=True, xticklabels=True, col_cluster=False, row_colors=row_colors)
    g.cax.set_visible(False)
    
    plt.setp(g.ax_heatmap.get_yticklabels(), fontsize='xx-small', fontstyle='italic')
    plt.show()

    return row_linkage, clusters

row_linkage, clusters = create_clustermap(adata=adata_subset, gene_list=gene_list, n_bins=500, figsize=(4,30), desired_n_clusters=3)




from collections import defaultdict

def create_cluster_dict(clusters, gene_list):
    """
    Creates a dictionary mapping cluster names to lists of genes, adjusting for zero-based indexing.

    Parameters:
    clusters (array): Array of cluster assignments for each gene.
    gene_list (list): List of genes corresponding to the clusters.

    Returns:
    dict: Dictionary with cluster names as keys and lists of genes as values.
    """
    cluster_dict = defaultdict(list)
    for gene, cluster in zip(gene_list, clusters):
        #print(f"Gene: {gene}, Cluster: {cluster}")  # Diagnostic print
        cluster_name=f"cluster_{cluster - 1}"
        cluster_dict[cluster_name].append(gene)

    return dict(cluster_dict)

# Example usage
cluster_dict = create_cluster_dict(clusters, gene_list=gene_list)

# Check the clusters and the number of genes in each
for cluster, genes in cluster_dict.items():
    print(f"{cluster}: {len(genes)} genes")

cluster_dict.keys()
scores=[]
for cluster_name,cluster_genes in cluster_dict.items():
    score_name=f"{cluster_name}_({len(cluster_genes)} genes)"
    scores.append(score_name)
    sc.tl.score_genes(adata_gex,gene_list=cluster_genes,score_name=f"{score_name}")

sc.pl.umap(adata_gex, color=scores, ncols=4, wspace=0.8, cmap="RdPu")








import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

optimal_orders = {}

for score in scores:
    # Prepare the data
    tmp_df = adata_gex.obs[["t", score]]
    X = tmp_df["t"].values.reshape(-1, 1)
    y = tmp_df[score].values

    # Split the data into training and testing sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    best_order = 1
    best_mse = float('inf')

    # Test different polynomial orders
    for order in range(1, 6):
        model = make_pipeline(PolynomialFeatures(order), LinearRegression())
        model.fit(X_train, y_train)
        predictions = model.predict(X_test)
        mse = mean_squared_error(y_test, predictions)

        if mse < best_mse:
            best_mse = mse
            best_order = order

    optimal_orders[score] = best_order






import matplotlib.pyplot as plt
import seaborn as sns

# Number of scores (subplots)
num_scores = len(scores)

# Create a figure with a row of subplots
fig, axes = plt.subplots(1, num_scores, figsize=(5 * num_scores, 5), sharey=True)

# Track the global min and max for the y-axis
global_min, global_max = float('inf'), float('-inf')

# Plot each score
for i, score in enumerate(scores):
    # Prepare the data
    tmp_df = adata_gex.obs[["t", score, "fine_grain"]]
    
    # Plot for each fine_grain category
    fine_grain_categories = tmp_df["fine_grain"].unique()
    for category in fine_grain_categories:
        category_data = tmp_df[tmp_df["fine_grain"] == category]
        sns.regplot(ax=axes[i], data=category_data, x="t", y=score, order=optimal_orders[score], scatter=False, label=category)
    
    # Set the title and add legend
    axes[i].set_title(score)
    axes[i].legend()

    # Update global min and max
    current_min, current_max = tmp_df[score].min(), tmp_df[score].max()
    global_min, global_max = min(global_min, current_min), max(global_max, current_max)

# Set the same y-axis limits for all subplots
for ax in axes:
    ax.set_ylim(global_min, global_max)

# Adjust layout
plt.tight_layout()
plt.show()
```

Example 141 (unknown):
```unknown
Creating clustermap for 29 features across 500 bins
```

Example 142 (unknown):
```unknown
Fitting expression...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 29/29 [00:02<00:00, 10.71it/s]
```

Example 143 (unknown):
```unknown
cluster_1: 8 genes
cluster_4: 5 genes
cluster_2: 4 genes
cluster_3: 10 genes
cluster_0: 2 genes
```

Example 144 (unknown):
```unknown
gene_set_B=cluster_dict["cluster_2"]+cluster_dict["cluster_1"]
gene_set_B
```

Example 145 (unknown):
```unknown
['DAB1',
 'UST',
 'ANGPT1',
 'GNAL',
 'KIF26B',
 'KCNQ5',
 'HEY1',
 'HSPA6',
 'RFTN1',
 'MAF',
 'OTOGL',
 'PDE1A']
```

Example 146 (python):
```python
sc.pl.umap(adata_gex, color=["milestones","seg"], ncols=4, wspace=0.8)
```

Example 147 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 148 (python):
```python
import pandas as pd

# Assuming adata_gex is your AnnData object containing scRNAseq data
# and it has 't' and 'milestones' in adata_gex.obs

# Filter the rows where 'milestones' is '60'
filtered_data = adata_gex.obs[adata_gex.obs['milestones'] == '60']

# Find the maximum 't' value in this filtered group
bifurcation_point = filtered_data['t'].median()

# Output the bifurcation point
print("Bifurcation Point:", bifurcation_point)
```

Example 149 (unknown):
```unknown
Bifurcation Point: 1.9802904538320618
```

Example 150 (python):
```python
import numpy as np
import pandas as pd

# Assuming adata_gex is your AnnData object containing scRNAseq data
# and bifurcation_point is already defined as shown previously

# Define the number of bins you want
X = 4  # Adjust this value as needed

# Get the minimum 't' value from adata_gex
min_t_value = adata_gex.obs['t'].min()

# Create bins between the minimum 't' value and the bifurcation point
bins = np.linspace(min_t_value, bifurcation_point, X + 1)  # X+1 because we need X bins

# Assign each cell to a pseudotime bin or "post_bifurcation"
adata_gex.obs['pseudotime_bin'] = pd.cut(adata_gex.obs['t'], bins=bins, include_lowest=True, 
                                         labels=[f'pseudotime_bin_{i}' for i in range(1, X + 1)])
adata_gex.obs['pseudotime_bin'] = adata_gex.obs['pseudotime_bin'].cat.add_categories('post_bifurcation')
adata_gex.obs.loc[adata_gex.obs['t'] > bifurcation_point, 'pseudotime_bin'] = 'post_bifurcation'

# You can now work with the 'pseudotime_bin' column in adata_gex.obs for further analysis
sc.pl.umap(adata_gex, color=["pseudotime_bin"], ncols=1, wspace=0.8, size=50)
```

Example 151 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 152 (python):
```python
adata_gex.obs.fine_grain.value_counts()
```

Example 153 (unknown):
```unknown
AtrialCardiomyocytesLeft     3041
AtrialCardiomyocytesRight    2623
Name: fine_grain, dtype: int64
```

Example 154 (python):
```python
import matplotlib
matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42
```

Example 155 (python):
```python
# Step 1: Copy values from 'fine_grain' to the new column
adata_gex.obs['AtrialCardiomyocytesImmature_annotation'] = adata_gex.obs['fine_grain']

# Add new category to the column
new_category = "AtrialCardiomyocytesImmature"
if new_category not in adata_gex.obs['AtrialCardiomyocytesImmature_annotation'].cat.categories:
    adata_gex.obs['AtrialCardiomyocytesImmature_annotation'] = adata_gex.obs['AtrialCardiomyocytesImmature_annotation'].cat.add_categories(new_category)

# Step 2: Modify values for cells not equal to "post_bifurcation" in 'pseudotime_bin'
adata_gex.obs.loc[adata_gex.obs['pseudotime_bin'] != 'post_bifurcation', 'AtrialCardiomyocytesImmature_annotation'] = new_category

# Verify the value counts
print(adata_gex.obs['AtrialCardiomyocytesImmature_annotation'].value_counts())
print("saving annotation for use with c2l...")
adata_gex.obs['AtrialCardiomyocytesImmature_annotation'].to_csv("AtrialCardiomyocytesImmature_annotation.csv")
sc.pl.umap(adata_gex,color="AtrialCardiomyocytesImmature_annotation",save="_AtrialCardiomyocytesImmature_annotation.pdf")
```

Example 156 (unknown):
```unknown
AtrialCardiomyocytesLeft        2828
AtrialCardiomyocytesRight       2509
AtrialCardiomyocytesImmature     327
Name: AtrialCardiomyocytesImmature_annotation, dtype: int64
saving annotation for use with c2l...
WARNING: saving figure to file figures/umap_AtrialCardiomyocytesImmature_annotation.pdf
```

Example 157 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 158 (python):
```python
# Add updated label - we have 2 young and 2 old populations

immature_barcodes = adata_gex.obs[adata_gex.obs['AtrialCardiomyocytesImmature_annotation']=="AtrialCardiomyocytesImmature"].index

# Create a dictionary to hold the updates
updates = {}

for barcode in immature_barcodes:
    if barcode in adata_gex.obs['fine_grain'].index:
        fine_grain_annotation = adata_gex.obs['fine_grain'][barcode]
        if fine_grain_annotation == 'AtrialCardiomyocytesLeft':
            updates[barcode] = 'AtrialCardiomyocytesLeft_immature'
        elif fine_grain_annotation == 'AtrialCardiomyocytesRight':
            updates[barcode] = 'AtrialCardiomyocytesRight_immature'

adata_gex.obs['AtrialCardiomyocytesImmature_annotation_v2'] = adata_gex.obs['AtrialCardiomyocytesImmature_annotation']

# First, add the new categories to the 'AtrialCardiomyocytesImmature_annotation_v2' column
new_categories = ['AtrialCardiomyocytesLeft_immature', 'AtrialCardiomyocytesRight_immature']
adata_gex.obs['AtrialCardiomyocytesImmature_annotation_v2'] = adata_gex.obs['AtrialCardiomyocytesImmature_annotation_v2'].cat.add_categories(new_categories)

# Then, you can apply the updates from the dictionary as before
for barcode, new_annotation in updates.items():
    adata_gex.obs.at[barcode, 'AtrialCardiomyocytesImmature_annotation_v2'] = new_annotation
    
# Remove unused categories from 'AtrialCardiomyocytesImmature_annotation_v2' column
adata_gex.obs['AtrialCardiomyocytesImmature_annotation_v2'] = adata_gex.obs['AtrialCardiomyocytesImmature_annotation_v2'].cat.remove_unused_categories()

# Check the categories after removal of unused ones
adata_gex.obs['AtrialCardiomyocytesImmature_annotation_v2'].value_counts()
```

Example 159 (unknown):
```unknown
AtrialCardiomyocytesLeft              2828
AtrialCardiomyocytesRight             2509
AtrialCardiomyocytesLeft_immature      213
AtrialCardiomyocytesRight_immature     114
Name: AtrialCardiomyocytesImmature_annotation_v2, dtype: int64
```

Example 160 (python):
```python
sc.pl.umap(adata_gex,color="AtrialCardiomyocytesImmature_annotation_v2",save="_AtrialCardiomyocytesImmature_annotation_v2.pdf")
```

Example 161 (unknown):
```unknown
WARNING: saving figure to file figures/umap_AtrialCardiomyocytesImmature_annotation_v2.pdf
```

Example 162 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 163 (python):
```python
adata_gex.X=adata_gex.layers["counts"].copy()
sc.pp.normalize_total(adata_gex, target_sum=1e4)
sc.pp.log1p(adata_gex)
```

Example 164 (python):
```python
WARNING: adata.X seems to be already log-transformed.
```

Example 165 (python):
```python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import scipy.stats
from scipy.sparse import issparse
from tqdm import tqdm
import scanpy as sc
import numpy as np

# Function to calculate the correlation matrix and scatter plot data
def calculate_correlation(bin_cells, gene_set_A, gene_set_B, gene_indices):
    combined_gene_set = gene_set_A + gene_set_B
    # Initialize an empty DataFrame for the correlation matrix
    correlation_matrix = pd.DataFrame(index=combined_gene_set, columns=combined_gene_set)
    scatter_data = []

    # Calculate correlations for each pair of genes
    for gene1 in tqdm(combined_gene_set):
        for gene2 in combined_gene_set:
            # Extract expression values for gene1 and gene2
            x_values = bin_cells.X[:, gene_indices[gene1]].toarray().flatten() if issparse(bin_cells.X) else bin_cells.X[:, gene_indices[gene1]].flatten()
            y_values = bin_cells.X[:, gene_indices[gene2]].toarray().flatten() if issparse(bin_cells.X) else bin_cells.X[:, gene_indices[gene2]].flatten()

            # Calculate Pearson correlation
            correlation, _ = scipy.stats.pearsonr(x_values, y_values)

            # Store the correlation in the matrix
            correlation_matrix.at[gene1, gene2] = correlation

    # Calculate mean correlation for each gene with gene_set_A and gene_set_B
    for gene in combined_gene_set:
        mean_corr_with_A = correlation_matrix.loc[gene, gene_set_A].mean()
        mean_corr_with_B = correlation_matrix.loc[gene, gene_set_B].mean()
        gene_set_label = 'Left aCM early module' if gene in gene_set_A else 'Right aCM early module'

        scatter_data.append({'gene': gene, 'mean_corr_with_A': mean_corr_with_A, 'mean_corr_with_B': mean_corr_with_B, 'gene_set': gene_set_label})

    scatter_df = pd.DataFrame(scatter_data)

    return scatter_df

# Function to add labels to the genes closest to the bottom right and top left of each scatter plot
def label_extreme_genes(scatter_df, ax):
    # Define bottom right and top left points
    bottom_right_point = (1, -1)
    top_left_point = (-1, 1)

    # Function to calculate Euclidean distance
    def euclidean_distance(point1, point2):
        return np.sqrt((point1[0] - point2[0])**2 + (point1[1] - point2[1])**2)

    # Find the gene closest to the bottom right
    scatter_df['distance_to_bottom_right'] = scatter_df.apply(lambda row: euclidean_distance((row['mean_corr_with_A'], row['mean_corr_with_B']), bottom_right_point), axis=1)
    bottom_right_gene = scatter_df.loc[scatter_df['distance_to_bottom_right'].idxmin()]

    # Find the gene closest to the top left
    scatter_df['distance_to_top_left'] = scatter_df.apply(lambda row: euclidean_distance((row['mean_corr_with_A'], row['mean_corr_with_B']), top_left_point), axis=1)
    top_left_gene = scatter_df.loc[scatter_df['distance_to_top_left'].idxmin()]

    # Add labels to the plot
    ax.text(bottom_right_gene['mean_corr_with_A'], bottom_right_gene['mean_corr_with_B'], bottom_right_gene['gene'], color='blue', ha='right')
    ax.text(top_left_gene['mean_corr_with_A'], top_left_gene['mean_corr_with_B'], top_left_gene['gene'], color='red', ha='left')

# Assuming gene_set_A and gene_set_B are lists of gene names
# Assuming adata_gex is your AnnData object containing scRNAseq data
gene_indices = {gene: adata_gex.var.index.get_loc(gene) for gene in set(gene_set_A + gene_set_B)}

# Get all unique categories from 'pseudotime_bin' and move 'post_bifurcation' to the end
bin_categories = [cat for cat in adata_gex.obs['pseudotime_bin'].cat.categories if cat != 'post_bifurcation']
bin_categories.append('post_bifurcation')

# Number of bins to create subplots for
num_bins = len(adata_gex.obs['pseudotime_bin'].cat.categories)

# Create a figure with two rows of subplots: one for UMAP and one for scatter plots
fig, axes = plt.subplots(2, num_bins, figsize=(6 * num_bins, 12))

# Loop over each pseudotime bin to create UMAP and scatter plots
for i, bin_label in enumerate(adata_gex.obs['pseudotime_bin'].cat.categories):
    # Skip the post_bifurcation bin if needed
    #if bin_label == 'post_bifurcation':
    #    continue

    # Filter cells in the current bin for scatter plot
    bin_cells = adata_gex[adata_gex.obs['pseudotime_bin'] == bin_label]

    # Calculate scatter plot data
    scatter_df = calculate_correlation(bin_cells, gene_set_A, gene_set_B, gene_indices)

    # Create UMAP plot in the first row
    ax_umap = axes[0, i]
    sc.pl.umap(adata_gex, color="pseudotime_bin", groups=[bin_label], ax=ax_umap, show=False, legend_loc=None, title="", size=50)

    # Create specific scatter plot in the second row
    ax_scatter = axes[1, i]
    sns.scatterplot(data=scatter_df, x='mean_corr_with_A', y='mean_corr_with_B', hue='gene_set', ax=ax_scatter)
    ax_scatter.set_title(f'{bin_label}')
    ax_scatter.axhline(0, color='grey', linestyle='--')
    ax_scatter.axvline(0, color='grey', linestyle='--')
    ax_scatter.set_xlabel('Correlation with Left aCM early module')
    ax_scatter.set_ylabel('Correlation with Right aCM early module')
    ax_scatter.set_xlim(-1, 1)
    ax_scatter.set_ylim(-1, 1)
    ax_scatter.legend(title='Gene Module')

    # Label the extreme genes
    label_extreme_genes(scatter_df, ax_scatter)

plt.tight_layout()
plt.savefig("./figures/bifurcation_analysis.pdf",bbox_inches="tight")
plt.show()
```

Example 166 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 17.03it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 19.90it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 20.41it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 21.44it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:58<00:00,  2.67s/it]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 167 (python):
```python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import scipy.stats
from scipy.sparse import issparse
from tqdm import tqdm
import scanpy as sc
import numpy as np

# Function to calculate the correlation matrix and scatter plot data
def calculate_correlation(bin_cells, gene_set_A, gene_set_B, gene_indices):
    combined_gene_set = gene_set_A + gene_set_B
    # Initialize an empty DataFrame for the correlation matrix
    correlation_matrix = pd.DataFrame(index=combined_gene_set, columns=combined_gene_set)
    scatter_data = []

    # Calculate correlations for each pair of genes
    for gene1 in tqdm(combined_gene_set):
        for gene2 in combined_gene_set:
            # Extract expression values for gene1 and gene2
            x_values = bin_cells.X[:, gene_indices[gene1]].toarray().flatten() if issparse(bin_cells.X) else bin_cells.X[:, gene_indices[gene1]].flatten()
            y_values = bin_cells.X[:, gene_indices[gene2]].toarray().flatten() if issparse(bin_cells.X) else bin_cells.X[:, gene_indices[gene2]].flatten()

            # Calculate Pearson correlation
            correlation, _ = scipy.stats.pearsonr(x_values, y_values)

            # Store the correlation in the matrix
            correlation_matrix.at[gene1, gene2] = correlation

    # Calculate mean correlation for each gene with gene_set_A and gene_set_B
    for gene in combined_gene_set:
        mean_corr_with_A = correlation_matrix.loc[gene, gene_set_A].mean()
        mean_corr_with_B = correlation_matrix.loc[gene, gene_set_B].mean()
        gene_set_label = 'Left aCM early module' if gene in gene_set_A else 'Right aCM early module'

        scatter_data.append({'gene': gene, 'mean_corr_with_A': mean_corr_with_A, 'mean_corr_with_B': mean_corr_with_B, 'gene_set': gene_set_label})

    scatter_df = pd.DataFrame(scatter_data)

    return scatter_df

# Function to add labels to the genes closest to the bottom right and top left of each scatter plot
def label_extreme_genes(scatter_df, ax):
    # Define bottom right and top left points
    bottom_right_point = (1, -1)
    top_left_point = (-1, 1)

    # Function to calculate Euclidean distance
    def euclidean_distance(point1, point2):
        return np.sqrt((point1[0] - point2[0])**2 + (point1[1] - point2[1])**2)

    # Find the gene closest to the bottom right
    scatter_df['distance_to_bottom_right'] = scatter_df.apply(lambda row: euclidean_distance((row['mean_corr_with_A'], row['mean_corr_with_B']), bottom_right_point), axis=1)
    bottom_right_gene = scatter_df.loc[scatter_df['distance_to_bottom_right'].idxmin()]

    # Find the gene closest to the top left
    scatter_df['distance_to_top_left'] = scatter_df.apply(lambda row: euclidean_distance((row['mean_corr_with_A'], row['mean_corr_with_B']), top_left_point), axis=1)
    top_left_gene = scatter_df.loc[scatter_df['distance_to_top_left'].idxmin()]

    # Add labels to the plot
    ax.text(bottom_right_gene['mean_corr_with_A'], bottom_right_gene['mean_corr_with_B'], bottom_right_gene['gene'], color='blue', ha='right')
    ax.text(top_left_gene['mean_corr_with_A'], top_left_gene['mean_corr_with_B'], top_left_gene['gene'], color='red', ha='left')

# Assuming gene_set_A and gene_set_B are lists of gene names
# Assuming adata_gex is your AnnData object containing scRNAseq data
gene_indices = {gene: adata_gex.var.index.get_loc(gene) for gene in set(gene_set_A + gene_set_B)}

# Get all unique categories from 'pseudotime_bin' and move 'post_bifurcation' to the end
bin_categories = [cat for cat in adata_gex.obs['pseudotime_bin'].cat.categories if cat != 'post_bifurcation']
bin_categories.append('post_bifurcation')

# Number of bins to create subplots for
num_bins = len(adata_gex.obs['pseudotime_bin'].cat.categories)

# Create a figure with two rows of subplots: one for UMAP and one for scatter plots
fig, axes = plt.subplots(2, num_bins, figsize=(6 * num_bins, 12))

# Loop over each pseudotime bin to create UMAP and scatter plots
for i, bin_label in enumerate(adata_gex.obs['pseudotime_bin'].cat.categories):
    # Skip the post_bifurcation bin if needed
    #if bin_label == 'post_bifurcation':
    #    continue

    # Filter cells in the current bin for scatter plot
    bin_cells = adata_gex[adata_gex.obs['pseudotime_bin'] == bin_label]

    # Calculate scatter plot data
    scatter_df = calculate_correlation(bin_cells, gene_set_A, gene_set_B, gene_indices)

    # Create UMAP plot in the first row
    ax_umap = axes[0, i]
    sc.pl.umap(adata_gex, color="pseudotime_bin", groups=[bin_label], ax=ax_umap, show=False, legend_loc=None, title="", size=50)

    # Create specific scatter plot in the second row
    ax_scatter = axes[1, i]
    sns.scatterplot(data=scatter_df, x='mean_corr_with_A', y='mean_corr_with_B', hue='gene_set', ax=ax_scatter)
    ax_scatter.set_title(f'{bin_label}')
    ax_scatter.axhline(0, color='grey', linestyle='--')
    ax_scatter.axvline(0, color='grey', linestyle='--')
    ax_scatter.set_xlabel('Correlation with Left aCM early module')
    ax_scatter.set_ylabel('Correlation with Right aCM early module')
    ax_scatter.set_xlim(-1, 1)
    ax_scatter.set_ylim(-1, 1)
    ax_scatter.legend(title='Gene Module')

    # Label the extreme genes
    label_extreme_genes(scatter_df, ax_scatter)

plt.tight_layout()
plt.savefig("./figures/bifurcation_analysis.pdf",bbox_inches="tight")
plt.show()
```

Example 168 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 19.79it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 20.46it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 21.47it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 21.43it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:58<00:00,  2.67s/it]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 169 (python):
```python
# Create a figure with two rows of subplots: one for UMAP and one for scatter plots with gene names
fig, axes = plt.subplots(2, num_bins, figsize=(6 * num_bins, 12))

# Loop over each pseudotime bin to create UMAP and scatter plots
for i, bin_label in enumerate(adata_gex.obs['pseudotime_bin'].cat.categories):
    # Filter cells in the current bin for scatter plot
    bin_cells = adata_gex[adata_gex.obs['pseudotime_bin'] == bin_label]

    # Calculate scatter plot data
    scatter_df = calculate_correlation(bin_cells, gene_set_A, gene_set_B, gene_indices)

    # Create UMAP plot in the first row
    ax_umap = axes[0, i]
    sc.pl.umap(adata_gex, color="pseudotime_bin", groups=[bin_label], ax=ax_umap, show=False, legend_loc=None, title=bin_label, size=50)

    # Create scatter plot in the second row with gene names
    ax_scatter = axes[1, i]
    # Instead of using sns.scatterplot, we loop through the scatter_df to place text
    for _, row in scatter_df.iterrows():
        ax_scatter.text(row['mean_corr_with_A'], row['mean_corr_with_B'], row['gene'], fontsize=9)
    
    ax_scatter.set_title(f'{bin_label}')
    ax_scatter.axhline(0, color='grey', linestyle='--')
    ax_scatter.axvline(0, color='grey', linestyle='--')
    ax_scatter.set_xlabel('Correlation with Left aCM early module')
    ax_scatter.set_ylabel('Correlation with Right aCM early module')
    ax_scatter.set_xlim(-0.5, 0.5)
    ax_scatter.set_ylim(-0.5, 0.5)

plt.tight_layout()
#plt.savefig("./figures/bifurcation_analysis.pdf",bbox_inches="tight")
plt.show()
```

Example 170 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 19.95it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 20.54it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 20.61it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 21.67it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:58<00:00,  2.68s/it]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
```

Example 171 (python):
```python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import scipy.stats
from scipy.sparse import issparse
from tqdm import tqdm
import scanpy as sc
import numpy as np

# Assuming the rest of your code remains the same...

# Create a figure with two rows of subplots: one for UMAP and one for scatter plots with gene names
fig, axes = plt.subplots(2, num_bins, figsize=(6 * num_bins, 12))

scatter_plot_axis_limit = 0.45

# Loop over each pseudotime bin to create UMAP and scatter plots
for i, bin_label in enumerate(adata_gex.obs['pseudotime_bin'].cat.categories):
    # Filter cells in the current bin for scatter plot
    bin_cells = adata_gex[adata_gex.obs['pseudotime_bin'] == bin_label]

    # Calculate scatter plot data
    scatter_df = calculate_correlation(bin_cells, gene_set_A, gene_set_B, gene_indices)

    # Create UMAP plot in the first row
    ax_umap = axes[0, i]
    sc.pl.umap(adata_gex, color="pseudotime_bin", groups=[bin_label], ax=ax_umap, show=False, legend_loc=None, title=bin_label, size=50)

    # Create scatter plot in the second row with gene names, colored by gene set
    ax_scatter = axes[1, i]
    for _, row in scatter_df.iterrows():
        # Determine the color based on the gene set
        color = 'blue' if row['gene'] in gene_set_A else 'orange'
        ax_scatter.text(row['mean_corr_with_A'], row['mean_corr_with_B'], row['gene'], fontsize=9, color=color, style='italic')
    
    ax_scatter.set_title(f'{bin_label}')
    ax_scatter.axhline(0, color='grey', linestyle='--')
    ax_scatter.axvline(0, color='grey', linestyle='--')
    ax_scatter.set_xlabel('Correlation with Left aCM early module')
    ax_scatter.set_ylabel('Correlation with Right aCM early module')
    ax_scatter.set_xlim(-scatter_plot_axis_limit, scatter_plot_axis_limit)
    ax_scatter.set_ylim(-scatter_plot_axis_limit, scatter_plot_axis_limit)

plt.tight_layout()
plt.savefig("./figures/bifurcation_analysis_text_names.pdf",bbox_inches="tight")
plt.show()
```

Example 172 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 19.19it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 20.37it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 21.41it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 21.60it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:55<00:00,  2.52s/it]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
```

Example 173 (python):
```python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import scipy.stats
from scipy.sparse import issparse
from tqdm import tqdm
import scanpy as sc
import numpy as np

# Assuming the rest of your code remains the same...

# Create a figure with two rows of subplots: one for UMAP and one for scatter plots with gene names
fig, axes = plt.subplots(2, num_bins, figsize=(6 * num_bins, 12))

scatter_plot_axis_limit = 0.3

# Loop over each pseudotime bin to create UMAP and scatter plots
for i, bin_label in enumerate(adata_gex.obs['pseudotime_bin'].cat.categories):
    # Filter cells in the current bin for scatter plot
    bin_cells = adata_gex[adata_gex.obs['pseudotime_bin'] == bin_label]

    # Calculate scatter plot data
    scatter_df = calculate_correlation(bin_cells, gene_set_A, gene_set_B, gene_indices)

    # Create UMAP plot in the first row
    ax_umap = axes[0, i]
    sc.pl.umap(adata_gex, color="pseudotime_bin", groups=[bin_label], ax=ax_umap, show=False, legend_loc=None, title=bin_label, size=50)

    # Create scatter plot in the second row with gene names, colored by gene set
    ax_scatter = axes[1, i]
    for _, row in scatter_df.iterrows():
        # Determine the color based on the gene set
        color = 'blue' if row['gene'] in gene_set_A else 'orange'
        ax_scatter.text(row['mean_corr_with_A'], row['mean_corr_with_B'], row['gene'], fontsize=9, color=color, style='italic')
    
    ax_scatter.set_title(f'{bin_label}')
    ax_scatter.axhline(0, color='grey', linestyle='--')
    ax_scatter.axvline(0, color='grey', linestyle='--')
    ax_scatter.set_xlabel('Correlation with Left aCM early module')
    ax_scatter.set_ylabel('Correlation with Right aCM early module')
    ax_scatter.set_xlim(-scatter_plot_axis_limit, scatter_plot_axis_limit)
    ax_scatter.set_ylim(-scatter_plot_axis_limit, scatter_plot_axis_limit)

plt.tight_layout()
#plt.savefig("./figures/bifurcation_analysis_text_names.pdf",bbox_inches="tight")
plt.show()
```

Example 174 (unknown):
```unknown
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 19.87it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 20.46it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 21.44it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
  0%|                                                                                                                                              | 0/22 [00:00<?, ?it/s]/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scipy/stats/_stats_py.py:4781: ConstantInputWarning: An input array is constant; the correlation coefficient is not defined.
  warnings.warn(stats.ConstantInputWarning(msg))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 21.70it/s]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:55<00:00,  2.51s/it]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
posx and posy should be finite values
```

Example 175 (unknown):
```unknown
early_modules_dict={}
early_modules_dict["LEFT"]=gene_set_A
early_modules_dict["RIGHT"]=gene_set_B
```

Example 176 (unknown):
```unknown
['MBP',
 'KCNMB2',
 'GAN',
 'SLC8A3',
 'A2M',
 'KCNMB2-AS1',
 'KSR1',
 'PANCR',
 'PITX2',
 'DCBLD2']
```

Example 177 (unknown):
```unknown
['DAB1',
 'UST',
 'ANGPT1',
 'GNAL',
 'KIF26B',
 'KCNQ5',
 'HEY1',
 'HSPA6',
 'RFTN1',
 'MAF',
 'OTOGL',
 'PDE1A']
```

Example 178 (python):
```python
import pickle

key="aCM_n570" # or (aCM_n1140 or aCM_full)
key="aCM_n1140"
key="aCM_full"
target_type="Gene"
# File path
file_path = f"/nfs/team205/heart/pyscenic/plus/eRegulon_signatures/{key}_{target_type}_eRegulon_signatures.pkl"

# Open the file and load the dictionary
with open(file_path, 'rb') as file:
    TF_TG_dict = pickle.load(file)
```

Example 179 (unknown):
```unknown
results_list=[]
```

Example 180 (python):
```python
side="LEFT"
associated_features_dict.keys()
total_transcriptome_genes=36000
genes_of_interest=early_modules_dict[f"{side}"]

import pandas as pd
from scipy.stats import fisher_exact


# Initialize a list to store the results
results = []

for regulon, regulon_genes in TF_TG_dict.items():
    # Contingency table
    overlap = len(set(regulon_genes) & set(genes_of_interest))
    only_genes_of_interest = len(genes_of_interest) - overlap
    only_regulon_genes = len(regulon_genes) - overlap
    neither = total_transcriptome_genes - (overlap + only_genes_of_interest + only_regulon_genes)
    table = [[overlap, only_genes_of_interest], [only_regulon_genes, neither]]

    # Perform Fisher's Exact Test
    odds_ratio, p_value = fisher_exact(table)
    
    # Append the results
    results.append({"Regulon": regulon, "Odds Ratio": odds_ratio, "pval": p_value})

# Convert the list of dictionaries to a DataFrame
results_df = pd.DataFrame(results)

# Sort the DataFrame by p-value
results_df.sort_values(by="pval", inplace=True)

# Reset index after sorting
results_df.reset_index(drop=True, inplace=True)

# Display the DataFrame
results_df=results_df[results_df.pval<0.01]
results_df["side"]=side
results_df=results_df.reset_index(names='rank')
results_list.append(results_df)
results_df
```

Example 181 (python):
```python
side="RIGHT"
associated_features_dict.keys()
total_transcriptome_genes=36000
genes_of_interest=early_modules_dict[f"{side}"]

import pandas as pd
from scipy.stats import fisher_exact


# Initialize a list to store the results
results = []

for regulon, regulon_genes in TF_TG_dict.items():
    # Contingency table
    overlap = len(set(regulon_genes) & set(genes_of_interest))
    only_genes_of_interest = len(genes_of_interest) - overlap
    only_regulon_genes = len(regulon_genes) - overlap
    neither = total_transcriptome_genes - (overlap + only_genes_of_interest + only_regulon_genes)
    table = [[overlap, only_genes_of_interest], [only_regulon_genes, neither]]

    # Perform Fisher's Exact Test
    odds_ratio, p_value = fisher_exact(table)
    
    # Append the results
    results.append({"Regulon": regulon, "Odds Ratio": odds_ratio, "pval": p_value})

# Convert the list of dictionaries to a DataFrame
results_df = pd.DataFrame(results)

# Sort the DataFrame by p-value
results_df.sort_values(by="pval", inplace=True)

# Reset index after sorting
results_df.reset_index(drop=True, inplace=True)

# Display the DataFrame
results_df=results_df[results_df.pval<0.01]
results_df["side"]=side
results_df=results_df.reset_index(names='rank')
results_list.append(results_df)
results_df
```

Example 182 (python):
```python
results_concat=pd.concat(results_list)
print(results_concat.shape)
results_concat["TF"]=results_concat.Regulon.str.split("_",expand=True)[0]
results_concat
```

Example 183 (python):
```python
chdgene_table=pd.read_csv("chdgene_table.csv")
results_concat=results_concat.merge(chdgene_table, left_on="TF",right_on="Gene", how='left')
results_concat
```

Example 184 (python):
```python
import matplotlib
matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42
```

Example 185 (python):
```python
# Transforming p-values to -log10(p)
results_concat['-log10(pval)'] = -np.log10(results_concat['pval'])

# Create a pivot table for the heatmap
heatmap_data = results_concat.pivot("TF", "side", "-log10(pval)")
heatmap_data = heatmap_data.fillna(value=0)
heatmap_data = heatmap_data.sort_values(by=["LEFT","RIGHT"], ascending=False)

# Initialize with empty strings
annotations = pd.DataFrame('', index=heatmap_data.index, columns=heatmap_data.columns)

# Loop through the original dataframe to populate the annotations
for _, row in results_concat.iterrows():
    tf = row['TF']
    side = row['side']
    chd_classification = row['CHD classification']
    if pd.notna(chd_classification):
        if "ASD" in chd_classification:
            annotations.loc[tf, side] = '**'  # Two asterisks if "ASD" in CHD classification
        else:
            annotations.loc[tf, side] = '*'   # One asterisk otherwise

# Create the heatmap
plt.figure(figsize=(10, 10))
sns.heatmap(heatmap_data, annot=annotations, fmt="s", cmap="RdPu", square=True, cbar=True)
plt.title("Enrichment of early gene modules for regulons\n '*': associated with any CHD, '**' associated with ASD\ncolor scale = -log10(p-value)")
plt.xlabel("Side")
plt.ylabel("Regulon (Transcription Factor)")
plt.savefig("CHD_enrichment.pdf", format='pdf', bbox_inches='tight')
plt.show()
```

Example 186 (unknown):
```unknown
/tmp/ipykernel_4090211/3468513932.py:5: FutureWarning: In a future version of pandas all arguments of DataFrame.pivot will be keyword-only.
  heatmap_data = results_concat.pivot("TF", "side", "-log10(pval)")
```

Example 187 (unknown):
```unknown
[reg for reg in list(TF_TG_dict.keys()) if "PITX" in reg]
```

Example 188 (unknown):
```unknown
['PITX2_+_(130g)']
```

Example 189 (unknown):
```unknown
plot.GO_plot(gene_list=TF_TG_dict["PITX2_+_(130g)"],display_table=True,gene_sets=['GO_Biological_Process_2023'])
```

Example 190 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 191 (python):
```python
sc.pl.dotplot(adata_gex, groupby="AtrialCardiomyocytesImmature_annotation",var_names=TF_TG_dict["PITX2_+_(130g)"])
```

Example 192 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_dotplot.py:749: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap', 'norm' will be ignored
  dot_ax.scatter(x, y, **kwds)
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/g2g_L_v_RaCM_AUC.html

**Contents:**
- Tutorial on single-cell trajectory alignment using Genes2Genes¶
- Make AUC objects¶
- Split into REF and QUERY¶
  - Min max normalize the pseudotime¶
  - Check the number of bins in the optimal binning structure using OptBinning package¶
  - Visualize the interpolation binning structure in terms of the cell type composition¶
  - Run G2G alignment¶
  - The alignment distribution across all genes¶
  - Run gene set overrepresentation analysis over the top k mismatching genes¶
  - Clustering genes using their alignments¶
  - Below is the Levenshtein distance heat map of all genes ordered based on the above clustering structure¶

G2G aims to guide downstream comparative analysis of single-cell reference and query systems along any axis of progression (e.g. differentiation pseudotime, disease/treatment response pseudotime etc.). This notebook describes how we can use G2G framework to infer and analyse gene-level trajectory alignments between a given reference and query dataset.

In this tutorial, we are going to compare trajectories between two treatment groups (PAM and LPS) of mouse bone marrow-derived dendritic cells from Shalek et al (2014). The single cell datasets and their pseudotime estimates were downloaded from https://github.com/shenorrLab/cellAlign (Alpert et al 2018) and packaged into adata objects. There are 2 gene modules: global (core antiviral module) and local (peaked inflammatory module) considered by Alpert et al (2018), and we use the local module as an example comparison.

Load the reference and query anndata objects

At this point we have 4 objects

each has a per cell value (expression or score) for different types of features, and each has the same cells (with psudotemporal ordering in t)

OptBinning estimates 14 optimal number of splits for both reference and query pseudotime distributions. Therefore we choose the same number of interpolation points.

For this dataset, we use the author-given time annotations (1h,2h,4h,6h) as the cell-type annotations.

Next we define a colormap of our choice for these annotations, and call the below function.

This is done by first creating an aligner object, passing and setting all relevant parameters. Next we align all gene pairs. (This step is parallelizing indepedenent gene-alignments to make the process time-efficient, however the computational time for an individual alignment will increase as the number of cells and/or the number of interpolation time points increase.

Now we can check the aggregate (average) alignment across all genes:

We can also visualize this alignment in terms of cell-type composition

We can also visualize an individual gene (e.g. JUNB), displaying its alignment statistics

We can use the alignment similarity percentage statistic of genes to rank genes from highly distant to highly similar

Let us use 30% alignment similarity (=0.3) as a threshold in this case

We first run cluster diagnostics to decide on a distance threshold with a good tradeoff between the number of clusters and the quality of structure. We use levenshtein distance metric

We can inspect structures of distance thresholds that give local optimals of the mean silhouette score.

If we select distance threshold 0.37 which gives a local optimal of 0.3749 silhouette_score with 7 clusters

Below visualizes all alignment paths in each cluster along with its number of genes

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import gseapy as gp

def GO_plot(gene_list, gene_sets=['MSigDB_Hallmark_2020', 'KEGG_2021_Human', 'GO_Biological_Process_2023', 'Reactome_2022'], 
            display_table=False, title_suffix=None, sort_by='Combined Score', n_rank = 10, n_genes=None, FDR=None):
    """
    Generate GO plots and optionally display tables based on enrichment analysis.

    Parameters:
    - gene_list: List of gene names as strings
    - gene_sets: List of gene set names as strings
    - display_table: Boolean flag to display table or not
    - title_suffix: Suffix for the plot title
    - sort_by: Column name to sort the results and plot. Other options in table, e.g. 'n_genes'
    - n_rank: number of terms to include in plot
    """
    
    # Perform enrichment analysis
    enr_res = gp.enrichr(gene_list=gene_list,
                         gene_sets=gene_sets,
                         organism='human',
                         outdir=None)
    
    # Loop through each gene set to process and plot the results
    for geneset in gene_sets:
        print(geneset)
        
        # Filter the results based on the Gene_set
        df = enr_res.results[enr_res.results['Gene_set'] == geneset]
        
        # Calculate the number of genes in the overlap and add as a new column
        df['n_genes'] = [int(x.split('/')[0]) for x in df['Overlap']]
        
        # Sort the dataframe based on the specified column
        df = df.sort_values(sort_by, ascending=False)

        # Only use terms where a threshold number of genes are present
        if n_rank is not None:
            df = df[df.n_genes>=n_genes]

        if FDR is not None:
            df = df[df['Adjusted P-value']<FDR]
        
        # Optionally display the table
        if display_table:
            display(df[:30])
        
        # Generate the bar plot for the top 10 ranked terms
        plt.rcParams['axes.grid'] = False
        plt.rcParams['figure.figsize'] = 4, 3
        plt.barh(width=df[:n_rank][sort_by],
                 y=[x.split(' (')[0] for x in df[:n_rank]['Term']],
                 color='darkred')
        plt.gca().invert_yaxis()
        plt.xlabel(sort_by)
        if title_suffix:
            plt.title(f'{geneset}_{title_suffix}')
        else:
            plt.title(f'{geneset}')
        plt.margins(y=0.02)
        plt.show()
        
    return df
```

Example 2 (python):
```python
import anndata

def split_adata_by_modality(adata, modality_column, modality_values):
    """
    Splits an AnnData object into multiple AnnData objects based on a given modality column.
    
    Parameters:
        adata (anndata.AnnData): The original AnnData object.
        modality_column (str): The name of the column in adata.var that contains the modality information.
        modality_values (list): List of unique modalities that you want to split the adata into.
        
    Returns:
        dict: A dictionary where the keys are the modality names and the values are the new AnnData objects.
    """
    # Initialize an empty dictionary to hold the new AnnData objects
    splitted_adatas = {}
    
    # Loop through each modality value to create a new AnnData object
    for modality in modality_values:
        # Get the indices of the features belonging to the current modality
        feature_idx = adata.var[modality_column] == modality
        
        # Create a new AnnData object with only the features belonging to the current modality
        new_adata = anndata.AnnData(X=adata[:, feature_idx].X,
                                    obs=adata.obs,
                                    var=adata.var.loc[feature_idx],
                                    obsm=adata.obsm,
                                    uns=adata.uns)
        
        # Add the new AnnData object to the dictionary
        splitted_adatas[modality] = new_adata
    
    return splitted_adatas
```

Example 3 (python):
```python
import scanpy as sc
import anndata
import numpy as np
import seaborn as sb
import matplotlib.pyplot as plt
import platform
from optbinning import ContinuousOptimalBinning
import pandas as pd


from genes2genes import Main
from genes2genes import VisualUtils
from genes2genes import ClusterUtils
from genes2genes import TimeSeriesPreprocessor
from genes2genes import PathwayAnalyserV2

print(platform.python_version())
```

Example 4 (python):
```python
import matplotlib
matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42
```

Example 5 (unknown):
```unknown
temporal_ordering='age_days_smoothed_once'
```

Example 6 (python):
```python
paired_adata_with_time_path = "/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA/aCMs_WholeSample_paired_with_time.h5ad"
```

Example 7 (python):
```python
adata=sc.read(paired_adata_with_time_path)
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
adata.obs["time"]=adata.obs['t'].copy()
```

Example 10 (python):
```python
adata.X.data[:5]
```

Example 11 (unknown):
```unknown
array([2., 1., 1., 1., 1.], dtype=float32)
```

Example 12 (python):
```python
sc.pl.umap(adata,color=["seg","milestones","t","fine_grain"],ncols=2)
```

Example 13 (python):
```python
modality_column = "modality"
modality_values = ["Gene Expression", "Peaks"]
splitted_adatas = split_adata_by_modality(adata, modality_column, modality_values)

adata_gex = splitted_adatas["Gene Expression"]
print(adata_gex)

adata_atac = splitted_adatas["Peaks"]
print(adata_atac)
```

Example 14 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 36601
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones', 'time'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
AnnData object with n_obs × n_vars = 5664 × 508040
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones', 'time'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
```

Example 15 (unknown):
```unknown
path_to_auc_gene="/nfs/team205/heart/pyscenic/plus/AUCscore/foetal_aCM_n570_eRegulon_AUC_gene.csv"
path_to_auc_region="/nfs/team205/heart/pyscenic/plus/AUCscore/foetal_aCM_n570_eRegulon_AUC_region.csv"
```

Example 16 (python):
```python
# read in regulon scores
auc_gene = pd.read_csv(path_to_auc_gene, index_col=0)
print(auc_gene.shape)
auc_region = pd.read_csv(path_to_auc_region, index_col=0)
print(auc_region.shape)
```

Example 17 (unknown):
```unknown
(17302, 278)
(12964, 278)
```

Example 18 (python):
```python
# make regulon adata, AUC gene
# AUC Gene
# create anndata
adata_aucg = anndata.AnnData(
    X=auc_gene.loc[adata_gex.obs_names],
    obs=adata_gex.obs,
    var=pd.DataFrame(auc_gene.columns).set_index(0),
    obsm=adata_gex.obsm,
    uns= adata_gex.uns
)
# modify varnames
adata_aucg.var_names = [x.split('_(')[0] for x in adata_aucg.var_names]
print(adata_aucg.shape)

# AUC Region
# create anndata
adata_aucr = anndata.AnnData(
    X=auc_region.loc[adata_gex.obs_names],
    obs=adata_gex.obs,
    var=pd.DataFrame(auc_region.columns).set_index(0),
    obsm=adata_gex.obsm
)
# modify varnames
adata_aucr.var_names = [x.split('_(')[0] for x in adata_aucr.var_names]
print(adata_aucr.shape)
```

Example 19 (unknown):
```unknown
(5664, 278)
(5664, 278)
```

Example 20 (python):
```python
# split intlo REF and QUERY
adata_ref=adata_aucg[adata_aucg.obs.fine_grain=="AtrialCardiomyocytesLeft"].copy()
adata_query=adata_aucg[adata_aucg.obs.fine_grain=="AtrialCardiomyocytesRight"].copy()
```

Example 21 (python):
```python
sc.pl.umap(adata_ref,color=["time","fine_grain"],ncols=2)
sc.pl.umap(adata_query,color=["time","fine_grain"],ncols=2)
```

Example 22 (python):
```python
print(adata_ref)
print(adata_query)
```

Example 23 (unknown):
```unknown
AnnData object with n_obs × n_vars = 3041 × 278
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones', 'time'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
AnnData object with n_obs × n_vars = 2623 × 278
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones', 'time'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
```

Example 24 (python):
```python
# check the current range
print(min(adata_ref.obs['time']), max(adata_ref.obs['time']))
print(min(adata_query.obs['time']), max(adata_query.obs['time']))

# if it does not follow [0,1] range, run below
adata_ref.obs['time'] = TimeSeriesPreprocessor.Utils.minmax_normalise(np.asarray(adata_ref.obs['time']))
adata_query.obs['time'] = TimeSeriesPreprocessor.Utils.minmax_normalise(np.asarray(adata_query.obs['time']))
```

Example 25 (unknown):
```unknown
0.4222375414474298 8.245970642037001
0.45019867983228623 8.303595486512195
```

Example 26 (python):
```python
# Visualize the pseudotime distributions
sb.kdeplot(adata_ref.obs['time'], fill=True, color='forestgreen') 
sb.kdeplot(adata_query.obs['time'], fill=True, color='midnightblue'); 
plt.xlabel('pseudotime'); plt.legend(); plt.show()
```

Example 27 (unknown):
```unknown
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
```

Example 28 (python):
```python
x = np.asarray(adata_ref.obs.time)
optb = ContinuousOptimalBinning(name='pseudotime', dtype="numerical")
optb.fit(x, x)
print(len(optb.splits))

x = np.asarray(adata_query.obs.time)
optb = ContinuousOptimalBinning(name='pseudotime', dtype="numerical")
optb.fit(x, x)
print(len(optb.splits))
```

Example 29 (python):
```python
col = np.array(sb.color_palette('colorblind'))[range(2)]
col
```

Example 30 (unknown):
```unknown
array([[0.00392157, 0.45098039, 0.69803922],
       [0.87058824, 0.56078431, 0.01960784]])
```

Example 31 (python):
```python
joint_cmap={'AtrialCardiomyocytesLeft':col[0], 'AtrialCardiomyocytesRight':col[1]}
vs = VisualUtils.VisualUtils.get_celltype_composition_across_time(adata_ref, adata_query, n_points=14, 
                                     ANNOTATION_COLNAME='fine_grain', optimal_binning=False)
```

Example 32 (unknown):
```unknown
# trying max n points for optimal binning = 14
====================================================
Optimal equal number of bins for R and Q =  14
```

Example 33 (python):
```python
gene_list = adata_ref.var_names
print(len(gene_list))
```

Example 34 (unknown):
```unknown
gene_list[:10]
```

Example 35 (unknown):
```unknown
Index(['ARID3A_extended_-', 'ARNT2_+', 'ARNTL_+', 'ARNT_+', 'ATF1_+', 'ATF2_+',
       'ATF3_+', 'ATF6_extended_+', 'ATF7_+', 'BCL6_+'],
      dtype='object')
```

Example 36 (python):
```python
%%time

import os
import pickle

# Path for the folder and file
folder_path = '/nfs/team205/jc48/projects/fetal/code/g2g_aligners'
file_path = os.path.join(folder_path, 'AUCgene_aligner.pkl')

# Create folder if it doesn't exist
os.makedirs(folder_path, exist_ok=True)

# Check if the file already exists
if os.path.isfile(file_path):
    # Read the aligner object from the file
    print("this specific aligner already exists, reading it in...")
    with open(file_path, 'rb') as file:
        aligner = pickle.load(file)
else:
    # If the file does not exist, create and configure the aligner
    print("aligner not already made, making it...")
    aligner = Main.RefQueryAligner(adata_ref, adata_query, gene_list, len(vs.optimal_bining_S))
    aligner.WEIGHT_BY_CELL_DENSITY = True
    aligner.WINDOW_SIZE = 0.1
    aligner.state_params = [0.99, 0.1, 0.7]
    aligner.optimal_binning = True
    aligner.opt_binning_S = vs.optimal_bining_S
    aligner.opt_binning_T = vs.optimal_bining_T

    # Align all pairs
    aligner.align_all_pairs()

    # Save the aligner object to the file
    with open(file_path, 'wb') as file:
        pickle.dump(aligner, file)
```

Example 37 (unknown):
```unknown
this specific aligner already exists, reading it in...
CPU times: user 646 ms, sys: 66.9 ms, total: 713 ms
Wall time: 751 ms
```

Example 38 (unknown):
```unknown
aligner.get_aggregate_alignment()
```

Example 39 (unknown):
```unknown
Average Alignment:  IIIDIDIDDIDDMMMMMMMM
```

Example 40 (unknown):
```unknown
vs.visualize_gene_alignment("IDDDMMMMMMMMMIIIDID")#, cmap=joint_cmap)
```

Example 41 (unknown):
```unknown
df = aligner.get_stat_df()
```

Example 42 (unknown):
```unknown
mean matched percentage: 
60.519999999999996 %
```

Example 43 (unknown):
```unknown
VisualUtils.plot_alignmentSim_vs_l2fc(df)
```

Example 44 (unknown):
```unknown
df[(df["alignment_similarity_percentage"]<0.5)&(df["abs_l2fc"]>0)].sort_values('alignment_similarity_percentage', ascending=True)
```

Example 45 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(10, 10))


#y_axis = 'opt_alignment_cost'
y_axis = 'alignment_similarity_percentage'

sns.scatterplot(data=df, x='l2fc', y=y_axis, edgecolor="none",legend=False, alpha=0.8)

# Add a horizontal line at y=0.5
plt.axhline(0.5, color='gray', linestyle='--', linewidth = 0.5)

# Add text annotations for LaCM and RaCM
plt.text(0.1, 1.05, 'RaCM', transform=plt.gca().transAxes, ha='left', va='top', weight='semibold')
plt.text(0.9, 1.05, 'LaCM', transform=plt.gca().transAxes, ha='right', va='top', weight='semibold')


# Set labels
plt.xlabel("Log2 Fold Change")

if y_axis == "alignment_similarity_percentage":
    plt.ylabel("Alignment Similarity (%)")

if y_axis == "opt_alignment_cost":
    plt.ylabel("Alignment Cost (nits)")

# Save the plot
#plt.savefig("similarity_v_logFC.pdf", format='pdf', bbox_inches='tight')

# Show the plot
plt.show()
```

Example 46 (python):
```python
query_color = 'orange'
ref_color = 'blue'

def plotTimeSeries_cmap(gene, aligner, plot_cells = False, plot_mean_trend= False):
    
        al_obj = aligner.results_map[gene]
        plt.subplots(1,3,figsize=(15,3))
        plt.subplot(1,3,1)
        plotTimeSeriesAlignment_cmap(gene, aligner) 
        plt.subplot(1,3,2)
        max_val = np.max([np.max(np.asarray(aligner.ref_mat[al_obj.gene])), np.max(np.asarray(aligner.query_mat[al_obj.gene]))])
        min_val = np.min([np.min(np.asarray(aligner.ref_mat[al_obj.gene])), np.min(np.asarray(aligner.query_mat[al_obj.gene]))])
        g = sb.scatterplot(x=aligner.query_time, y=np.asarray(aligner.query_mat[al_obj.gene]), alpha=0.7, color = query_color, legend=False,linewidth=0.3, s=20)  
        plt.title('Query')
        plt.ylim([min_val-0.5,max_val+0.5])
        plt.subplot(1,3,3)
        g = sb.scatterplot(x=aligner.ref_time, y=np.asarray(aligner.ref_mat[al_obj.gene]), color = ref_color, alpha=0.7, legend=False,linewidth=0.3,s=20 ) 
        plt.title('Reference')
        plt.ylim([min_val-0.5,max_val+0.5])

def plotTimeSeriesAlignment_cmap(gene, aligner):  
    
        al_obj = aligner.results_map[gene]
        sb.scatterplot(x=al_obj.S.X, y=al_obj.S.Y, color = ref_color ,alpha=0.05, legend=False)#, label='Ref') 
        sb.scatterplot(x=al_obj.T.X, y=al_obj.T.Y, color = query_color ,alpha=0.05, legend=False)#, label ='Query') 
        plot_mean_trends(al_obj)
        plt.title(al_obj.gene)
        plt.xlabel('pseudotime')
        plt.ylabel('Gene expression')
        plt.axis('off')
        
        for i in range(al_obj.matched_region_DE_info.shape[0]):
            S_timebin = int(al_obj.matched_region_DE_info.iloc[i]['ref_bin'])
            T_timebin = int(al_obj.matched_region_DE_info.iloc[i]['query_bin']) 
            x_vals = [al_obj.matched_region_DE_info.iloc[i]['ref_pseudotime'],al_obj.matched_region_DE_info.iloc[i]['query_pseudotime']] 
            y_vals = [al_obj.S.mean_trend[S_timebin ], al_obj.T.mean_trend[T_timebin]] 
            plt.plot(x_vals, y_vals, color='black', linestyle='dashed', linewidth=1.5)

def plot_mean_trends(al_obj):
    # Plotting the mean trends for S and T with specified colors
    al_obj.S.plot_mean_trend(color=ref_color)
    al_obj.T.plot_mean_trend(color=query_color)
```

Example 47 (python):
```python
import matplotlib.pyplot as plt
from contextlib import redirect_stdout
import io


for gene in ['ETS2_-','ETS2_+','ETV6_+','GATA5_+','GATA4_+','THRA_+','PITX2_+','NKX2-5_extended_+']:
    print(gene)
    # Redirect the output of the plot function to a StringIO object
    f = io.StringIO()
    with redirect_stdout(f):
        #VisualUtils.plotTimeSeries(gene, aligner, vs)
        #VisualUtils.plotTimeSeriesAlignment(gene, aligner)
        plotTimeSeriesAlignment_cmap(gene, aligner)

    
    # Now, the plot is created and can be saved
    plt.savefig(f"{gene}_time_series_alignment.png", format='png', bbox_inches='tight')
    plt.show()
```

Example 48 (unknown):
```unknown
NKX2-5_extended_+
```

Example 49 (unknown):
```unknown
lowSimGenes = df[list(df['alignment_similarity_percentage'] < 0.5)]
topSimGenes = df[list(df['alignment_similarity_percentage'] > 0.35)]
```

Example 50 (unknown):
```unknown
# Running experiment to determine the distance threshold with a good trade-off
df = ClusterUtils.run_clustering(aligner, metric='levenshtein', experiment_mode=True)
```

Example 51 (unknown):
```unknown
compute distance matrix
using levenshtein distance metric
```

Example 52 (unknown):
```unknown
74%|██████████████████████████████████████████████████████████████████████████████████████████████████                                   | 73/99 [00:00<00:00, 82.46it/s]
```

Example 53 (unknown):
```unknown
df.sort_values('Distance threshold', ascending=False)
```

Example 54 (unknown):
```unknown
aligner.show_ordered_alignments()
```

Example 55 (unknown):
```unknown
Gene               Alignment
-----------------  ----------------------------
ETS2_+             IIIDIDIDIDIIIIIIIIDDDDDDDDDD
ETS2_-             IIIDIDIDIDIIIIIIIIDDDDDDDDDD
ETV6_+             IIIDIDIDIDIIIIIIIIDDDDDDDDDD
MAF_-              IIIDIDIDIDIIIIIIIIDDDDDDDDDD
PITX2_+            IIIDIDIDIDIIIIIIIIDDDDDDDDDD
E4F1_+             MMMMMMMMMMMMMM
FOXN2_+            MMMMMMMMMMMMMM
IRF7_extended_+    MMMMMMMMMMMMMM
MLX_+              MMMMMMMMMMMMMM
NFYA_+             MMMMMMMMMMMMMM
PHF21A_extended_+  MMMMMMMMMMMMMM
TBP_+              MMMMMMMMMMMMMM
TFAP4_+            MMMMMMMMMMMMMM
TFCP2_+            MMMMMMMMMMMMMM
TFDP2_+            MMMMMMMMMMMMMM
ZBTB20_+           MMMMMMMMMMMMMM
ZNF200_+           MMMMMMMMMMMMMM
ZNF709_+           MMMMMMMMMMMMMM
ZNF526_+           MMWWWWWWIIIIIMMMMMIID
ZNF337_+           MVVVVVVDDDDDDMMMMMMM
PATZ1_+            MMMMMMIDIDMMMMMM
ZNF267_+           MMMVVVDIDDDMMMMMMM
ZNF639_+           MMMMMMIDIDMMMMMM
ZNF394_+           MMMMMIDMMMMMMMM
ZNF676_+           MMMMMIDIDIDMMMMMM
ARNT2_+            MMMMIIDDIDMMMMMMM
ARNT_+             MMMMDDMMMMMMMMII
CEBPZ_+            MMMMDDMMMMMMMMII
CREB3_+            MMMMIIDDMMMMMMMM
FOS_+              MMMMIDMMMMMMMMM
GATA4_+            MMMMIDIDIDMMMMMMM
GATA4_-            MMMMIIDDIDMMMMMMM
GLIS3_-            MMMMIDDMMMMMIMMM
HES1_+             MMMMDDMMMMMMMMVV
HES6_extended_+    MMMMIDIDIDMMMMMMM
HMGA1_+            MMMMDDMMMMMMMMII
IRF3_+             MMMMIDIDMMMMMMMM
KLF11_+            MMMMDDMMMMMMMIIM
KLF2_+             MMMMDDMMMMMMMMVV
PHF20_+            MMMMDIDMMMMMMMMI
SALL2_+            MMMMIDIDMMMMMMMM
SIX4_extended_+    MMMMIIDDMMMMMMMM
TCFL5_+            MMMMIDMMMMMMMMM
ZBTB41_extended_+  MMMMDDMMMMMMMMVV
ZFP90_+            MMMMIIDDIDMMMMMMM
ZNF134_+           MMMMDDMMMMMMMMVV
ZNF135_+           MMMMDDMMMMMMMMVV
ZNF142_+           MMMMIIDDMMMMMMMM
ZNF429_+           MMMMDDMMMMMMMMVV
ZNF552_+           MMMMIIDDMMMMMMMM
ZNF571_+           MMMMIDIDIDMMMMMMM
ZNF610_+           MMMMIDIDMMMMMMMM
ZNF91_+            MMMMDDMMMMMMMMVV
ATF1_+             MMMDDDDMMIIIIMMMMM
ATF3_+             MMMIDDIDIDMMMMMIMM
CREBZF_extended_+  MMMIDIDIDMMMMMMMM
ELF2_+             MMMDIDDMMMMMMMMII
ETV5_+             MMMDDDMMMMMMMMIII
GMEB1_+            MMMIIDDMMMMMMMMM
IRF2_+             MMMDDIDMMMMMMMMVV
JUNB_+             MMMIDMMMMMMMMMM
MECP2_+            MMMDDIDMMMMMMMMII
SRF_+              MMMIDDDMMMMMMMMVV
TEAD2_+            MMMDDDMMMMMMMMVVV
TEAD4_+            MMMIDIDIDIDDMMMMMIID
TFDP1_+            MMMDDDMMMMMMMMVVV
ZBTB11_+           MMMDDDMMMMMMMMIII
ZBTB25_+           MMMDDMMMMMMMMMII
ZBTB5_+            MMMIDIDIDIDIDMMMMMM
ZFP62_+            MMMIIIDDDIDMMMMMMM
ZNF143_+           MMMDDMMMMMMMMMII
ZNF260_+           MMMIDIDIDMMMMMMMM
ZNF646_+           MMMDDDMMMMMMMMVVV
ZNF721_+           MMMDDDMMMMMMMMIII
HAND2_+            MMDMIIDIDIDIDMMMMMM
JUN_+              MMIIDDDDIDIDMMMMMMII
MXD4_+             MMIIIMMDDIDIDDMMMMM
NR3C1_+            MMDDDMMMMMMMMMIII
SMAD3_+            MMDDDDDDMMIIIIMMMIIID
STAT5B_+           MMIIDDMMMMMMMMMM
TBX20_+            MMDDDDIDIDIDMMMMMIIII
ZBTB1_extended_+   MMDDIDMMMMMMMMMVV
ZBTB7B_+           MMDIDIDDMMMMMMIIMM
ZFP14_+            MMIDIDIDIIDDMMMMMMM
ZNF121_+           MMIMMDIDMMMMMMMM
ZNF441_+           MMDDDDMMMIMMMMMIII
ZNF517_+           MMDDDMMMMMMMMMVVV
ZNF566_+           MMDDDDMMIIIIMMMMMM
ZNF641_+           MMIIDIDIDIDIDDMMMMMM
ATF2_+             MDDDDMMMIIIIMMMMMM
BHLHE41_+          MIIIVVVVIIDIIIIDDDDDDDDDDDD
CARF_+             MIIMMMDDMMMMMMMM
CREB1_+            MIIMMDDMMMMMMMMM
NCOA1_+            MIIIMMMDDDMMMMIIIDDD
OVOL2_+            MIIMMDDMMMMMMMMM
SOX4_+             MDDMMIDIDIDIDDMMMMIII
TFEB_+             MIIIIMMMDIDDDDMMMMM
ZNF235_+           MDMMDMMMMMMMMMVV
ZNF454_+           MIIMMMDDMMMMMMMM
ZNF518A_+          DMMMMMMMMMMMMMI
MEIS1_+            DMMMMMMMMMMMIMM
GLIS2_+            IMMMMMDMMMMMMMM
LIN28B_+           IMMMMDIDDMMMMMIMM
RFX2_+             IMMMMDIDIDMMMMMMM
TCF7L1_+           IMMMMDIDMMMMMMMM
ZNF44_+            IMMMMDIDMMMMMMMM
IRF9_+             IMMMDDMMMMMMMIMM
MEF2A_+            IMMMDDDMMMMMMIIMM
NFIC_-             DMMMIIDIDMMMMMMMM
CUX1_+             IMMDIDDDMMMMMMIIMM
FOSL2_+            IMMIIDIDIIDDDDDDMMMMII
NFE2L2_+           IMMDDDMMMMMMMMMII
PLAGL1_extended_+  DMMDIDIDMMMMMMMMVV
PLAGL2_+           IMMDDDDMMMMMMMMVVV
RARA_+             DMMDDMMMMMMMMMIII
RARG_+             IMMDIDDDMMMMMMMMVV
ZBTB10_+           DMMIDIDIDIIIIDDDMMMMM
ZNF117_+           IMMDDMMMMMMMMIMM
ZNF28_+            IMMDDDDMMMMMMMMVVV
ZNF248_+           IDMMMMMMMMMMMMM
ZNF273_+           IDMMMMMMMMMMMMM
ZNF777_+           IDMMMMMMMMMMMMM
ZNF579_+           DDMMMMMMMMMMMMII
STAT4_+            DIVVVVVVDDDDDDMMMMMMM
CREB3L4_+          IIMMMMDDMMMMMMMM
EGR1_+             IIMMMMDDMMMMMMMM
MTF1_+             IIMMMMDDIDMMMMMMM
PEG3_+             DDMMMMIDIDIIMMMMMM
SP1_+              IIMMMMDDMMMMMMMM
STAT2_+            IIMMMMDDMMMMMMMM
ZNF671_+           IIMMMMDDMMMMMMMM
ZNF814_+           IIMMMMDIDDMMMMMMM
E2F4_+             IDMMMDDMMMMMMMMVV
EGR2_+             IDMMMIDIDMMMMMMMM
JDP2_extended_+    DDMMMIIIDIDMMMMMMM
KLF13_+            IIMMMDIDDMMMMMMMM
KLF15_+            IDMMMDDMMMMMMMMII
NFYC_+             IDMMMDDMMMMMMIIMM
NR2C2_+            IDMMMDDMMMMMMMMII
ZFHX2_+            IIMMMDIDIDDMMMMMMM
ZNF281_+           IDMMMDDMMMMMMMMVV
ZNF557_+           IDMMMDDMMMMMMMMVV
ZSCAN18_+          IDMMMDIDMMMMMIMMM
CREB5_+            IDMMIDIDIDDDDMMMMMIII
MEIS2_+            IDMMIIDIDIDDMMMMMMM
MXD1_+             IDMMIDIDIIDDMMMMMMM
NFIB_+             IDMMIIDIDDMMMMMMMM
PBX1_+             DDMMIIDIDIIIDDMMMMMM
ZGPAT_+            DDMMIIDIDIMMMMMMMM
ZNF382_+           IIMMDDMMMMMMMMMM
ZSCAN30_+          IIMMDIDDDDMMMMMMMVV
KLF10_+            DDMIDIDIDMMMMMMMMVV
PRRX1_+            DDMIDIDIDIIMMMMMMMM
ZNF564_+           IDDMMMMMMMMMMMMI
CREB3L2_+          IDDMMMMMMMMMMIMM
ETV1_+             IDDMMMMMMMMMMIMM
ZNF654_+           IDDMMMMMMMMMMIMM
ZBTB8A_+           IDDMMMMMMIMMMMMM
BHLHE40_+          IDDMMMMIIIDDMMMMMM
MAZ_+              IIDMMMMIIDDDMMMMMM
ZNF644_+           IIIMMMMDDWWWIIMMMMM
ZNF780A_+          IIIMMMMDDDMMMMMMM
ZNF783_+           IIIMMMMDDDMMMMMMM
FOSB_+             IIDMMMDMMMMMMMMM
POU2F1_+           IDDMMMIMMMMMMMMM
PRDM10_+           IIDMMMIDDMMMMMMMM
RFX1_+             IDDMMMIIDIDMMMMMMM
RXRG_+             IDDMMMIIDMMMMMMMM
SP4_+              IIIMMMDDDMMMMMMMM
TCF7L2_+           IDDMMMIDIIDMMMMMMM
ZBTB44_+           IDDMMMIIDIIDDMMMMMM
ZNF530_+           IDDMMMIIDMMMMMMMM
ARNTL_+            IDDMMDDDMIIMMMMMMII
ATF7_+             IDDMMIDIDMMMMMIMMM
GMEB2_+            IDDMMDMMMMMMMMMII
KLF15_-            IDDMMIIDIDDDMMMMMMVV
MAX_+              IDDMMDDMMMMMMMMVVV
MBD2_+             IIDMMDDMMMMMMMMMI
NFE2L1_+           IDDMMIIDIDIIDDMMMMMID
NFIA_+             IDDMMIIDMMMMMMMMM
NKX2-5_extended_+  IDDMMDDMMMMMMMMVVV
NPAS2_extended_+   IDDMMDIDMMMMMMMMVV
PRDM6_+            IDDMMIIDIDMMMMMMMM
ZNF23_+            DDDMMIIIDIMMMMMMMM
ZNF25_+            IDDMMDMMMMMMMMMII
ZNF385D_-          IDDMMDIDMMMMMMIIMM
ZNF407_+           IDDMMDDMMMMMMMMIII
ZNF444_+           IIDMMDDDMMMMMMMIIM
ZNF561_+           IDDMMDDMMMMMMMMVVV
ZNF629_+           IDDMMDIDMMMMMMMMII
ZNF35_+            IDDMIDIDIIIDDMMMMMMM
ZNF362_+           DDDMIIDIDIIMMMMMMMM
MYCN_-             IDIDMMMMMMMMMMMM
ELF1_+             DDDDMMMMMIMMMMMIII
TCF3_+             DDDIVVVVVDDDMMMMMMMM
ZNF354C_+          DDDDMMMMMIIIIMMMMM
KLF3_+             IIIDMMMMDDMMMMMMM
RXRB_+             IIIDMMMMDDMMMMMMM
ZNF425_+           IIIDMMMMDDMMMMMMM
ZNF730_+           DDDIVVVVIIDDDDMMMMMMM
ATF6_extended_+    IIIIMMMIIDIDDDDDDMMMM
BCL6_+             IIIDMMMDDMMMMMMMM
KLF16_+            IIIDMMMIDIDDDMMMMMM
NFATC1_+           DDIDMMMIIMMMMMMMM
YY1_+              IDIDMMMIIDIDDMMMMMM
ZBTB7A_+           IDIDMMMIDIIDDMMMMMM
ZNF100_+           IIIDMMMIDDDMMMMMMM
ZNF214_+           IIIDMMMDDMMMMMMMM
ZNF511_+           IDIDMMMIDIIDDMMMMMM
ZNF846_-           IIIDMMMIDDDMMMMMMM
SOX5_extended_+    IDIDMMDDDMMMMMMMIII
ZNF48_+            IDIDMMDDMMMMMMMMVV
ZNF512_+           IDIDMMIIDDMMMMMMMM
ZNF470_+           DDIDMIIDIIIDDMMMMMMM
ZNF711_+           DDDIDMMMMMMMMMMVVV
ZBTB16_+           IDDIDMMMMMMMMMIMM
IRF4_+             DDDDDMMMMMIIIMMMMII
SREBF2_+           DDDDDMMMMIIMMMMMVVV
GTF2IRD1_+         IIIIIMMMIDIIIIIDDDDDDDDDD
MGA_+              IIDIDMMMDMMMMMMMM
PURA_+             DDDDDMMMIIIIIMMMMMM
RELA_+             IIIDIMMMIDDDDMMMMMM
SATB1_+            IIIDIMMMDDDMMMMMMM
VEZF1_+            IIIDIMMMDDDMMMMMMM
ZBTB43_extended_+  IIIDIMMMDDDMMMMMMM
ZNF555_+           DDDIDMMMIIIIDMMMMMM
ZNF770_+           IIDIDMMMIDIDDMMMMMM
FOXJ2_+            DDDIDMMIIIMMMMMMMM
HMGA1_-            IIDIDMMDDMMMMMMIMM
THRB_+             IDDIDMMIMMMMMMMMM
ZNF223_+           IIIDIMMDDDMMMMMMMM
IKZF2_+            DDDDDMIIDIIIIIDDDDMMIIIID
TRPS1_+            IDDIDMDIDMMMMMMMMII
ELK4_+             IDIDIDMMMMMMMMMMM
CEBPD_+            IIIDIDMMMDDMMMMMMM
FOS_-              DDDDDDMMMIIIIMMMMMII
MESP1_extended_+   DDDDDDMMMIIIIIIMMMMM
NR2F1_+            DDDIDDMMMIIIIMMMMMM
TEAD3_+            DDDDDDMMMIIIMMMMMIII
GBX1_extended_+    IIIDIDMMDDMMMMMMMM
MYCN_+             IIIIIIMMDDDDDMMMMMMD
THRA_+             IIIDIIMMDDDDMMMMMMM
ZNF432_+           IIIIIIMMDDDDDDMMMMMM
ZNF43_+            IIIDIDMMDDMMMMMMMM
ZNF503_+           DDDIDDDMMMIIIIIMMMMM
MAF_+              IIIIIIIMMIIDDDDDMMMDDDD
SMAD9_+            IIDIDIDMIIIDDDDMMMMMM
ZNF737_-           IIDIDIDMIDDMMMMMMMM
PRDM16_+           DDIDIDIDMMMMMMMMMVV
E2F5_+             IDIDIDIDMMMMMMMMMM
MAFG_+             IDIDIDIDMMMMMMMMMM
MBD1_+             DDDDDDDIVVVVVIMMMMMMM
ZNF596_+           DDDIDDDDMMMIIIIMMMIIID
ZBED6_extended_+   DDDIDIDIDMMMMMMMMVVV
ZNF202_+           DDDIDIDIDMMMMMMMMVVV
MEF2B_+            DDDDDDDDIVVVVVVIMMMMMM
SREBF1_+           DDIDIDIDIDMMMMMMMMVV
TGIF1_+            DDIDIDIDIDMMMMMMMMVV
MEF2A_-            IDIDIDIDIDMMMMMMMMM
ZSCAN9_+           IDIDIDIDIDMMMMMMMMM
NFIC_+             DDIDIDIDIDMMMMMMMMII
ZNF383_-           IDIDIDIDIDMMMMMMIDMM
ZNF207_+           DDIDIDIDIDDMMMMMMMVVV
HMGA2_+            IDIDIDIDIDDMMMMMMMMI
RUNX1_+            DDIDIDIDIDDMMMMMMMIII
ARID3A_extended_-  IIIDIDIDIDDDMMMMMMMM
MEF2C_+            IIIDIDIDIDDDMMMMMMMM
ZNF148_+           IIIDIDIDIDDDMMMMMMMM
ZNF540_+           IIIDIDIDIDDDMMMMMMMM
ZNF778_+           IIIDIDIDIDDDMMMMMMMM
KLF6_+             IIIDIDIDIDIDDDMMMMMMM
ZNF462_+           IIIDIDIDIDIDDDMMMMMMM
ZNF669_+           IIIDIDIDIDIDDDMMMMMMM
E2F6_+             IIIDIDIDIDIIDDDDMMMMMM
ZNF219_+           IIIDIDIDIDIIDDDDMMMMMM
ZNF549_+           IIIDIDIDIDIIDDDDMMMMMM
SIX5_extended_+    IIIDIDIDIDIIIDDDDDMMMMM
GATA5_+            IIIDIDIDIDIIDDDDDDDMMIIIID
```

Example 56 (unknown):
```unknown
31              ETS2_+
32              ETS2_-
35              ETV6_+
78               MAF_-
119            PITX2_+
            ...       
24              E2F6_+
203           ZNF219_+
242           ZNF549_+
139    SIX5_extended_+
44             GATA5_+
Name: genes, Length: 278, dtype: object
```

Example 57 (unknown):
```unknown
ClusterUtils.run_clustering(aligner, metric='levenshtein', DIST_THRESHOLD=0.5)
```

Example 58 (unknown):
```unknown
compute distance matrix
using levenshtein distance metric
run agglomerative clustering |  0.5
silhouette_score:  0.1830240565803321
```

Example 59 (unknown):
```unknown
ClusterUtils.visualise_clusters(aligner,n_cols = 4, figsize= (10,6))
```

Example 60 (unknown):
```unknown
ClusterUtils.print_cluster_average_alignments(aligner)
```

Example 61 (unknown):
```unknown
cluster:  0 IIIIIMMMIDDDDDDMMMMM 7
cluster:  1 IDMMMIIDDMMMMMMMM 234
cluster:  2 IIIDIDIDIDIIIIIIDDDDDDDMMD 3
cluster:  3 IIDDDDDDMMMIIIMVMMMM 19
cluster:  4 MIIIMMMIIDDDMMMMMDD 1
cluster:  5 DMDDDDMMIIIIIIMMMMMD 1
cluster:  6 IIIDIMIMDDDDMMMMMMM 4
cluster:  7 IIIDDDDDDMMVVMMMVMMM 2
cluster:  8 IIIDIDIDIDIIIIIIIIDDDDDDDDDD 7
```

Example 62 (unknown):
```unknown
# Initialize an empty dictionary
cluster_genes_dict = {}

# Loop through each gene and its corresponding cluster ID
for gene, cluster_id in zip(aligner.gene_list, aligner.cluster_ids):
    # Check if the cluster ID is already in the dictionary
    if cluster_id not in cluster_genes_dict:
        # If not, create a new list for this cluster ID
        cluster_genes_dict[cluster_id] = []
    # Append the gene to the list for this cluster ID
    cluster_genes_dict[cluster_id].append(gene)
```

Example 63 (unknown):
```unknown
clus_of_interest=8
```

Example 64 (python):
```python
cluster_genes_dict[clus_of_interest]
print(len(cluster_genes_dict[clus_of_interest]))
```

Example 65 (unknown):
```unknown
GENE_SUBSET = cluster_genes_dict[clus_of_interest]
aligner.get_aggregate_alignment_for_subset(GENE_SUBSET)
```

Example 66 (unknown):
```unknown
Average Alignment:  IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 67 (python):
```python
for gene in GENE_SUBSET:
    if len(GENE_SUBSET)<150:
        print(gene, aligner.results_map[gene].alignment_str)
        VisualUtils.plotTimeSeries(gene, aligner, vs)
        plt.show()
```

Example 68 (unknown):
```unknown
BHLHE41_+ MIIIVVVVIIDIIIIDDDDDDDDDDDD
```

Example 69 (unknown):
```unknown
ETS2_+ IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 70 (unknown):
```unknown
ETS2_- IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 71 (unknown):
```unknown
ETV6_+ IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 72 (unknown):
```unknown
GTF2IRD1_+ IIIIIMMMIDIIIIIDDDDDDDDDD
```

Example 73 (unknown):
```unknown
MAF_- IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 74 (unknown):
```unknown
PITX2_+ IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 75 (python):
```python
import matplotlib
matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42
```

Example 76 (python):
```python
import matplotlib.pyplot as plt
from contextlib import redirect_stdout
import io


for gene in GENE_SUBSET:
    f = io.StringIO()
    with redirect_stdout(f):
        plotTimeSeries_cmap(gene, aligner)
    plt.savefig(f"./figures/{gene}_time_series_alignment.png", format='png', bbox_inches='tight')
    plt.show()
```

Example 77 (python):
```python
for gene in GENE_SUBSET:
    sc.pl.umap(adata_aucg,color=gene,cmap="RdPu",save=f"_{gene}_AUC.png")
```

Example 78 (unknown):
```unknown
WARNING: saving figure to file figures/umap_BHLHE41_+_AUC.png
```

Example 79 (unknown):
```unknown
WARNING: saving figure to file figures/umap_ETS2_+_AUC.png
```

Example 80 (unknown):
```unknown
WARNING: saving figure to file figures/umap_ETS2_-_AUC.png
```

Example 81 (unknown):
```unknown
WARNING: saving figure to file figures/umap_ETV6_+_AUC.png
```

Example 82 (unknown):
```unknown
WARNING: saving figure to file figures/umap_GTF2IRD1_+_AUC.png
```

Example 83 (unknown):
```unknown
WARNING: saving figure to file figures/umap_MAF_-_AUC.png
```

Example 84 (unknown):
```unknown
WARNING: saving figure to file figures/umap_PITX2_+_AUC.png
```

Example 85 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt
from scipy.stats import pearsonr

# Assuming plot_data is defined as per your code snippet
plot_data = adata_aucg.obs[["t", "age_days_smoothed_once"]]

# Calculate Pearson's R and the p-value
pearson_r, p_value = pearsonr(plot_data['age_days_smoothed_once'], plot_data['t'])
p_value_text = f"{p_value:.2e}" if p_value > 1.0e-8 else "< 1.0e-8"

# Create scatter plot with specified aesthetics
sns.regplot(data=plot_data, x="age_days_smoothed_once", y="t", 
            scatter_kws={'alpha':0.5, 's':20, 'edgecolor':None},
            line_kws={'color':'black'}, # Sets the regression line color to black
            ci=95)

# Add text annotation for Pearson's R and adjusted p-value on the plot
plt.text(x=min(plot_data['age_days_smoothed_once']), y=max(plot_data['t']),
         s=f"Pearson's R: {pearson_r:.2f}\nP-value: {p_value_text}",
         backgroundcolor='white', ha='left', va='top')  # Adjust text positioning as needed
plt.xlabel("age (days)")
plt.ylabel("pseudotime")
plt.title('Pseudotime correlates with sample age')
plt.tight_layout()
plt.savefig(f"./figures/age_pseudotime_correlation.pdf", format='pdf', bbox_inches='tight')
plt.show()
```

Example 86 (unknown):
```unknown
df = aligner.get_stat_df()
```

Example 87 (unknown):
```unknown
mean matched percentage: 
60.519999999999996 %
```

Example 88 (unknown):
```unknown
df["alignment_similarity_percentage"] *= 100
```

Example 89 (unknown):
```unknown
filtered_df = df[df["alignment_similarity_percentage"] < 25]

# Sort primarily by 'l2fc' and secondarily by 'and_logfc', both in ascending order
sorted_df = filtered_df.sort_values(by=['l2fc'])
sorted_df
```

Example 90 (unknown):
```unknown
LOW_SIM_REGULONS = sorted_df['Gene'].unique().tolist()
```

Example 91 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(4, 4))

# Assuming 'df' is your DataFrame and 'genes_of_interest' is your list of genes
genes_of_interest = GENE_SUBSET

# Choose the y-axis based on your need
y_axis = 'alignment_similarity_percentage'  # or 'opt_alignment_cost'

# Plotting the scatter plot
sns.scatterplot(data=df, x='l2fc', y=y_axis, edgecolor="none", legend=False, alpha=0.3, color='black')
sns.scatterplot(data=df[df["Gene"].isin(LOW_SIM_REGULONS)], x='l2fc', y=y_axis, edgecolor="none", legend=False, color='red', alpha=0.8)

# Add a horizontal line at y=0.5
plt.axvline(0, color='gray', linestyle='--', linewidth=0.5)

# Add text annotations for LaCM and RaCM
plt.text(0.1, 1.05, 'RaCM', transform=plt.gca().transAxes, ha='left', va='top', weight='semibold')
plt.text(0.9, 1.05, 'LaCM', transform=plt.gca().transAxes, ha='right', va='top', weight='semibold')

# Label points corresponding to specific genes
#for _, row in df.iterrows():
#    if row['Gene'] in genes_of_interest:
#        plt.text(row['l2fc'], row[y_axis], row['Gene'], color='red', style='italic')

# Set labels
plt.xlabel("Log2 Fold Change")

if y_axis == "alignment_similarity_percentage":
    plt.ylabel("Alignment Similarity (%)")
elif y_axis == "opt_alignment_cost":
    plt.ylabel("Alignment Cost (nits)")

# Save the plot
plt.savefig("./figures/similarity_v_logFC_AUC.pdf", format='pdf', bbox_inches='tight')

# Show the plot
plt.show()
```

Example 92 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(6, 6))

# Assuming 'df' is your DataFrame and 'genes_of_interest' is your list of genes
genes_of_interest = GENE_SUBSET  # Make sure this is defined as your subset of interest

# Choose the y-axis based on your need
y_axis = 'alignment_similarity_percentage'  # or 'opt_alignment_cost'

# Plotting the background scatter plot
sns.scatterplot(data=df[~df["Gene"].isin(LOW_SIM_REGULONS)], x='l2fc', y=y_axis, edgecolor="none", legend=False, alpha=0.3, color='black')
sns.scatterplot(data=df[df["Gene"].isin(LOW_SIM_REGULONS)], x='l2fc', y=y_axis, edgecolor="none", legend=False, color='red', alpha=0.8)

# Instead of plotting red points, we will annotate gene names directly on the plot for LOW_SIM_REGULONS
for _, row in df[df["Gene"].isin(LOW_SIM_REGULONS)].iterrows():
    plt.text(row['l2fc']+0.01, row[y_axis]+1, row['Gene'], color='red', style='italic', alpha=0.8, size='x-small')

# Add a horizontal line at y=0
plt.axvline(0, color='gray', linestyle='--', linewidth=0.5)

# Add text annotations for LaCM and RaCM
plt.text(0.1, 1.05, 'RaCM', transform=plt.gca().transAxes, ha='left', va='top', weight='semibold')
plt.text(0.9, 1.05, 'LaCM', transform=plt.gca().transAxes, ha='right', va='top', weight='semibold')

# Set labels
plt.xlabel("Log2 Fold Change")
if y_axis == "alignment_similarity_percentage":
    plt.ylabel("Alignment Similarity (%)")
elif y_axis == "opt_alignment_cost":
    plt.ylabel("Alignment Cost (nits)")

# Save the plot
plt.savefig("./figures/similarity_v_logFC_AUC.pdf", format='pdf', bbox_inches='tight')

# Show the plot
plt.show()
```

Example 93 (python):
```python
sc.pl.matrixplot(adata_aucg,
             var_names = LOW_SIM_REGULONS,
             groupby="fine_grain",
             colorbar_title="Mean regulon\nactivity score",
                save="_RAS_LvRaCMs.pdf",
                cmap="RdPu",
                vmax=0.2,
                var_group_positions=[(0,1),(2,10)],
             var_group_labels=["RaCM","LaCM"])
```

Example 94 (unknown):
```unknown
WARNING: saving figure to file figures/matrixplot__RAS_LvRaCMs.pdf
```

Example 95 (python):
```python
adata_aucg.obs['t'] = pd.to_numeric(adata_aucg.obs['t'])
adata_aucg = adata_aucg[adata_aucg.obs['t'].sort_values().index]
```

Example 96 (python):
```python
sc.pl.heatmap(adata_aucg,var_names = LOW_SIM_REGULONS,
             groupby="fine_grain",
             standard_scale='var',
             cmap='RdPu',
              var_group_positions=[(0,1),(2,10)],
             var_group_labels=["RaCM","LaCM"],
              save="_RAS_LvRaCMs.pdf",
             )
```

Example 97 (unknown):
```unknown
WARNING: saving figure to file figures/heatmap_RAS_LvRaCMs.pdf
```

Example 98 (python):
```python
import scipy
# Extract the expression data for these genes
expression_data = adata_aucg[:, LOW_SIM_REGULONS].X

# Depending on the AnnData storage, the expression data might need conversion to a dense format
if scipy.sparse.issparse(expression_data):
    expression_data = expression_data.toarray()

# Create a DataFrame with the expression data
df_expression = pd.DataFrame(expression_data, columns=LOW_SIM_REGULONS)

# Add the pseudotime to the DataFrame
df_expression['pseudotime'] = adata_aucg.obs['t'].values

# Optionally, add your 'fine_grain' cluster information if you want to plot clusters in order
df_expression['fine_grain'] = adata_aucg.obs['fine_grain'].values

# Melt the DataFrame for easier plotting with seaborn
df_melted = df_expression.melt(id_vars=['pseudotime', 'fine_grain'], var_name='Gene', value_name='Expression')

# Plotting with seaborn
plt.figure(figsize=(10, 8))
sns.heatmap(df_melted.pivot_table(index='Gene', columns='pseudotime', values='Expression'), cmap='RdPu')

plt.title('Gene Expression and Pseudotime Heatmap')
plt.xlabel('Pseudotime')
plt.ylabel('Gene')
plt.show()
```

Example 99 (python):
```python
import matplotlib.pyplot as plt
from contextlib import redirect_stdout
import io


for gene in LOW_SIM_REGULONS:
    f = io.StringIO()
    with redirect_stdout(f):
        plotTimeSeries_cmap(gene, aligner)
    plt.savefig(f"./figures/{gene}_time_series_alignment.png", format='png', bbox_inches='tight')
    plt.show()
```

Example 100 (unknown):
```unknown
LOW_SIM_REGULONS
```

Example 101 (unknown):
```unknown
['ETS2_+',
 'MAF_+',
 'IKZF2_+',
 'ETV6_+',
 'BHLHE41_+',
 'GTF2IRD1_+',
 'GATA5_+',
 'PITX2_+',
 'SIX5_extended_+',
 'MAF_-',
 'ETS2_-']
```

Example 102 (python):
```python
for gene in LOW_SIM_REGULONS:
    sc.pl.umap(adata_aucg,color=gene,cmap="RdPu",save=f"_{gene}_AUC.png")
```

Example 103 (unknown):
```unknown
WARNING: saving figure to file figures/umap_ETS2_+_AUC.png
```

Example 104 (unknown):
```unknown
WARNING: saving figure to file figures/umap_MAF_+_AUC.png
```

Example 105 (unknown):
```unknown
WARNING: saving figure to file figures/umap_IKZF2_+_AUC.png
```

Example 106 (unknown):
```unknown
WARNING: saving figure to file figures/umap_ETV6_+_AUC.png
```

Example 107 (unknown):
```unknown
WARNING: saving figure to file figures/umap_BHLHE41_+_AUC.png
```

Example 108 (unknown):
```unknown
WARNING: saving figure to file figures/umap_GTF2IRD1_+_AUC.png
```

Example 109 (unknown):
```unknown
WARNING: saving figure to file figures/umap_GATA5_+_AUC.png
```

Example 110 (unknown):
```unknown
WARNING: saving figure to file figures/umap_PITX2_+_AUC.png
```

Example 111 (unknown):
```unknown
WARNING: saving figure to file figures/umap_SIX5_extended_+_AUC.png
```

Example 112 (unknown):
```unknown
WARNING: saving figure to file figures/umap_MAF_-_AUC.png
```

Example 113 (unknown):
```unknown
WARNING: saving figure to file figures/umap_ETS2_-_AUC.png
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/no3_create-network_TF-Region-TG_compact-common_up.html

**Contents:**
- Read in inputs¶
- Read in gene signatures and define target genes¶
- Read in peak signatures¶
- Read in scenicplus output metadata¶
- Read in and prepare anndata contains gene expression¶
- Read in cardiomyocyte markers¶
- Obtain expressing genes and accessible regions¶
- Genes¶
- Peaks¶
- Prepare scenic metadata¶
- Select interactions¶
- TF-TF network¶
- Calculate centrality¶
- Save networks¶
- Plot network¶
- Correlation of peak cound and n_TF to bind¶

What this notebook does

**Examples:**

Example 1 (python):
```python
# Parameters
working_dir = "/nfs/team205/heart/Vitro/TF_selection"
path_to_adata_gex = '/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.h5ad'
path_to_adata_peak_tfidf = "/nfs/team205/heart/anndata_objects/Foetal/multiome_ATAC/ArchR/project_output/PeakMatrix/Foetal_Peaks_tfidf.h5ad"
group_key = "t_binned"
group_a = "t3"
group_b = "t1"
time_key = "t"
subset_object = False
geneset_list = ["MSigDB_Hallmark_2020", "KEGG_2021_Human", "GO_Biological_Process_2023"]

path_to_eRegulon_signatures_gene = "/nfs/team205/heart/pyscenic/plus/eRegulon_signatures/vCM_onlyLeft_Gene_eRegulon_signatures.pkl"
path_to_eRegulon_signatures_region = "/nfs/team205/heart/pyscenic/plus/eRegulon_signatures/vCM_onlyLeft_Region_eRegulon_signatures.pkl"

path_to_DAR_compact = '/nfs/team205/heart/anndata_objects/Foetal/multiome_ATAC/ArchR/project_output/MarkerPeaks/vCM_Left_Compact-vs-Trab.csv'
path_to_DAR_trab = '/nfs/team205/heart/anndata_objects/Foetal/multiome_ATAC/ArchR/project_output/MarkerPeaks/vCM_Left_Trab-vs-Compact.csv'
path_to_eRegulon_metadata = "/nfs/team205/heart/pyscenic/plus/eRegulon_metadata/vCM_onlyLeft_eRegulon_metadata_filtered.csv" # filtered metadata

path_to_gene_signatures = '/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/gene_signatures.pkl'
path_to_peak_signatures = '/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/peak_signatures.pkl'

###### parameters for TF selection for the population ######
cell_of_intest = 'VentricularCardiomyocytesLeftCompact'
target_signatures = ['compact_specific_up','common_up']

# proportion threshold for expressing genes
exp_prop_thresh = 0.1

# proportion threshold for accesible regions
access_prop_thresh = 0.01
```

Example 2 (python):
```python
import scanpy as sc
import pandas as pd
import numpy as np
import anndata

import matplotlib.pyplot as plt
import seaborn as sns

import os
```

Example 3 (python):
```python
import session_info
session_info.show()
```

Example 4 (python):
```python
-----
anndata             0.9.2
matplotlib          3.7.2
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 5 (python):
```python
PIL                         10.0.0
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.11.17
cffi                        1.15.1
charset_normalizer          3.2.0
colorama                    0.4.6
comm                        0.1.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.8
decorator                   5.1.1
defusedxml                  0.7.1
executing                   1.2.0
fastjsonschema              NA
fqdn                        NA
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_metadata          NA
importlib_resources         NA
ipykernel                   6.25.1
isoduration                 NA
jedi                        0.19.0
jinja2                      3.1.2
joblib                      1.3.2
json5                       NA
jsonpointer                 2.0
jsonschema                  4.19.0
jsonschema_specifications   NA
jupyter_events              0.7.0
jupyter_server              2.7.1
jupyterlab_server           2.24.0
kiwisolver                  1.4.5
leidenalg                   0.10.1
llvmlite                    0.40.1
markupsafe                  2.1.3
mpl_toolkits                NA
natsort                     8.4.0
nbformat                    5.9.2
numba                       0.57.1
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.16.1
pyparsing                   3.0.9
pythonjsonlogger            NA
pytz                        2023.3
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
ruamel                      NA
scipy                       1.10.1
send2trash                  NA
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
sphinxcontrib               NA
stack_data                  0.6.2
statsmodels                 0.14.0
texttable                   1.6.7
threadpoolctl               3.2.0
tornado                     6.3.3
traitlets                   5.9.0
typing_extensions           NA
uri_template                NA
urllib3                     2.0.4
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.2
yaml                        6.0.1
zipp                        NA
zmq                         25.1.1
zstandard                   0.19.0
```

Example 6 (unknown):
```unknown
-----
IPython             8.12.0
jupyter_client      8.3.0
jupyter_core        4.12.0
jupyterlab          4.0.5
notebook            7.0.2
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-5.4.0-169-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-03-25 12:27
```

Example 7 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/regulon_plot.py")
regulon_plot = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = regulon_plot
spec.loader.exec_module(regulon_plot)
```

Example 8 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 9 (unknown):
```unknown
gene_signatures = utils.read_pkl(path_to_gene_signatures)
gene_signatures.keys()
```

Example 10 (unknown):
```unknown
dict_keys(['compact_specific_up', 'trab_specific_up', 'common_up', 'compact_specific_down', 'trab_specific_down', 'common_down'])
```

Example 11 (python):
```python
sig_genes = []
for sig in target_signatures:
    print(sig)
    sig_genes = sig_genes + gene_signatures[sig]
print(len(sig_genes))
sig_genes[:5]
```

Example 12 (unknown):
```unknown
compact_specific_up
common_up
466
```

Example 13 (unknown):
```unknown
['KIDINS220', 'ANKRD18A', 'MT-ATP6', 'ZNF385D', 'VEGFA']
```

Example 14 (unknown):
```unknown
peak_signatures = utils.read_pkl(path_to_peak_signatures)
peak_signatures.keys()
```

Example 15 (unknown):
```unknown
dict_keys(['compact_up', 'trab_up', 'compact_down', 'trab_down'])
```

Example 16 (python):
```python
sig_peaks = peak_signatures['compact_up']
print(len(sig_peaks))
sig_peaks[:5]
```

Example 17 (unknown):
```unknown
['chr1:240234383_240234883',
 'chr9:93673338_93673838',
 'chr22:42772683_42773183',
 'chr12:48124648_48125148',
 'chr12:46438700_46439200']
```

Example 18 (python):
```python
# scenicplus output
scenic_meta = pd.read_csv(path_to_eRegulon_metadata,index_col=0)
scenic_meta['Region'] = [x.replace('-','_') for x in scenic_meta['Region']]

# select 
scenic_meta.head()
```

Example 19 (python):
```python
# read in adata
# whole data
adata = sc.read_h5ad(path_to_adata_gex)

# subset gene expression data
adata = adata[:,adata.var['modality']=='Gene Expression']

# filter and log-normalise
sc.pp.filter_genes(adata, min_cells=3)
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

print(adata.X.data[:10])
adata
```

Example 20 (python):
```python
/nfs/team205/kk18/miniconda3/envs/vitro/lib/python3.8/site-packages/scanpy/preprocessing/_simple.py:251: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var['n_cells'] = number
```

Example 21 (unknown):
```unknown
[1.0619993 1.5653387 1.8985853 1.5653387 2.347629  1.8985853 1.5653387
 1.0619993 1.0619993 1.0619993]
```

Example 22 (unknown):
```unknown
AnnData object with n_obs × n_vars = 16451 × 26187
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N', 'n_cells'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'umap', 'week_colors', 'log1p'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
```

Example 23 (python):
```python
# read in major TF list
markers = pd.read_csv('./Gene_markers_Maturation.csv')
print(markers.shape)
markers.head()

# get non-TF markers (since this will overlap with centrality calculation based on TF-TF network)
marker_genes = list(markers[markers['Category']!='TF']['Gene'])
print(f'number of marker genes: {len(marker_genes)}')

# known TFs
TFs_known = list(markers[markers['Category']=='TF']['Gene'])
print(f'number of known TFs: {len(TFs_known)}')
TFs_known
```

Example 24 (unknown):
```unknown
(44, 4)
number of marker genes: 35
number of known TFs: 9
```

Example 25 (unknown):
```unknown
['MEF2A',
 'MEF2C',
 'TBX20',
 'NKX2-5',
 'HEY2',
 'PRDM16',
 'PPARA',
 'RORA',
 'THRA']
```

Example 26 (python):
```python
# get expression proportions
exp_mat = utils.get_exp_proportion(adata,groupby='fine_grain', use_rep='X')
exp_genes = list(exp_mat.index[exp_mat[cell_of_intest]>exp_prop_thresh])
len(exp_genes)
```

Example 27 (unknown):
```unknown
'PRDM16' in exp_genes
```

Example 28 (python):
```python
adata_peak = sc.read_h5ad(path_to_adata_peak_tfidf)
adata_peak
```

Example 29 (unknown):
```unknown
AnnData object with n_obs × n_vars = 167022 × 508040
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels'
    var: 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N'
```

Example 30 (python):
```python
# get accessible proportions
access_mat = utils.get_exp_proportion(adata_peak,groupby='fine_grain', use_rep='X')
```

Example 31 (unknown):
```unknown
access_peaks = list(access_mat.index[access_mat[cell_of_intest]>access_prop_thresh])
len(access_peaks)
```

Example 32 (python):
```python
# add columns to scenicplus metadata
# scenic_meta['Gene_isTF'] = scenic_meta['Gene'].isin(list(set(scenic_meta['TF'])))
scenic_meta['TF_isExp'] = scenic_meta['TF'].isin(exp_genes)
scenic_meta['TF_isSignature'] = (scenic_meta['TF'].isin(sig_genes))&(scenic_meta['TF'].isin(exp_genes))
scenic_meta['Region_isAccess'] = scenic_meta['Region'].isin(access_peaks)
scenic_meta['Region_isSignature'] = (scenic_meta['Region'].isin(sig_peaks))&(scenic_meta['Region'].isin(access_peaks))
scenic_meta.head()
```

Example 33 (python):
```python
# save the modified metadata
scenic_meta.to_csv(os.path.join(table_dir,'csvs/scenicplus_eRegulon_metadata_vCM_onlyLeft_modified.csv'))
```

Example 34 (python):
```python
os.path.join(table_dir,'csvs/scenicplus_eRegulon_metadata_vCM_onlyLeft_modified.csv')
```

Example 35 (python):
```python
'/nfs/team205/heart/fetal_ms_figures/tables/csvs/scenicplus_eRegulon_metadata_vCM_onlyLeft_modified.csv'
```

Example 36 (python):
```python
target_genes = sig_genes.copy()
print(len(target_genes))
target_genes[:5]
# removing TFs from the list
```

Example 37 (unknown):
```unknown
['KIDINS220', 'ANKRD18A', 'MT-ATP6', 'ZNF385D', 'VEGFA']
```

Example 38 (python):
```python
# ionchannel = list(pd.read_csv('/nfs/team205/kk18/data/geneset/HGNC/HGNC_GID177_Ion-channels.txt',sep='\t')['Approved symbol'].unique())
# gpcr = list(pd.read_csv('/nfs/team205/kk18/data/geneset/HGNC/HGNC_GID139_G-protein-coupled-receptors.txt',sep='\t')['Approved symbol'].unique())
```

Example 39 (python):
```python
# select interactions

# select interaction which target "target_genes"
scenic_meta_sel = scenic_meta[(scenic_meta['Gene'].isin(target_genes))&\
                               scenic_meta['TF_isExp'] &\
                               scenic_meta['Region_isAccess'] &\
                              (scenic_meta['TF_isSignature']|scenic_meta['Region_isSignature'])
                             ]

# get TFs and target genes in the selected interaction
direct_TFs = list(set(scenic_meta_sel['TF']))
targets = list(set(scenic_meta_sel['Gene']))
# remove TFs already in target genes
TFs_sel = list(set(direct_TFs) - set(targets))

print(len(TFs_sel))
print(f'direct TFs: {direct_TFs}')
print('')

# select higher level interactions 
print('Start looking for higher level TFs')
higher_level_TFs = []
while len(TFs_sel)>0:
    scenic_meta_sel2 = scenic_meta[(scenic_meta['Gene'].isin(TFs_sel))&\
                               scenic_meta['TF_isExp'] &\
                               scenic_meta['Region_isAccess'] &\
                              (scenic_meta['TF_isSignature']|scenic_meta['Region_isSignature'])
                                  ]
    higher_level_TFs = higher_level_TFs + list(set(scenic_meta_sel2['TF']))
    if len(scenic_meta_sel2)>0:
        # concatenate
        scenic_meta_sel = pd.concat([scenic_meta_sel,scenic_meta_sel2])
        scenic_meta_sel = scenic_meta_sel.drop_duplicates()
        # get TFs and target genes in the selected interaction
        TFs_sel = list(set(scenic_meta_sel['TF']))
        targets = list(set(scenic_meta_sel['Gene']))
        # remove TFs already in target genes
        TFs_sel = list(set(TFs_sel) - set(targets))
        print(f'{len(TFs_sel)}: {TFs_sel}')
    else:
        print('no more higher level TFs')
        TFs_sel = []

scenic_meta_sel['direct_TF'] = scenic_meta_sel['TF'].isin(direct_TFs)
higher_level_TFs = list(set(higher_level_TFs)-set(direct_TFs))

# reindex
scenic_meta_sel.index = range(len(scenic_meta_sel))

print("")
print(f'Number of selected interactions: {len(scenic_meta_sel)}')
print(f"Number of TFs: {len(set(scenic_meta_sel['TF']))}")
```

Example 40 (unknown):
```unknown
44
direct TFs: ['TFCP2', 'ZNF280D', 'AHCTF1', 'MEIS2', 'NFATC2', 'ZBTB20', 'UBP1', 'NPAS2', 'KLF12', 'SREBF2', 'MAX', 'HMBOX1', 'ESRRB', 'NR3C1', 'HAND2', 'CREB1', 'PHF21A', 'SOX5', 'ZBTB16', 'THRA', 'MEF2D', 'RORA', 'TGIF2', 'ARID3A', 'KLF6', 'MXI1', 'PLAG1', 'BHLHE40', 'ESRRA', 'NFIB', 'MEF2A', 'FOXK1', 'GTF2IRD1', 'BACH2', 'GATA6', 'MAZ', 'ZNF141', 'NCOA1', 'SMAD3', 'NFIC', 'HIF1A', 'NR4A1', 'LCORL', 'FOXN3', 'ATF1', 'GATA4', 'MEF2C', 'NFE2L1', 'MBNL2', 'YY1', 'HEY2', 'PRDM16', 'ZNF654', 'PPARA', 'TBX20', 'ZNF148', 'ZSCAN18', 'USF2', 'NFIA', 'PBX1', 'FOXO1', 'CUX1']

Start looking for higher level TFs
12: ['MEF2C', 'MAZ', 'AHCTF1', 'IRX3', 'POU2F1', 'ZNF141', 'ELF2', 'SMAD3', 'UBP1', 'HIF1A', 'ZNF586', 'FOXO1']
9: ['MEF2C', 'MAZ', 'AHCTF1', 'POU2F1', 'ZNF141', 'SMAD3', 'UBP1', 'HIF1A', 'FOXO1']
no more higher level TFs

Number of selected interactions: 4832
Number of TFs: 66
```

Example 41 (python):
```python
# get TF-TG dictionary
network = {}
for tf in set(scenic_meta_sel['TF']):
    df_sub = scenic_meta_sel[scenic_meta_sel['TF']==tf]
    network[tf] = list(set(df_sub['Gene']))
print(len(network.keys()))
```

Example 42 (unknown):
```unknown
# signature genes which are not included in the interaction
# just checking
genes_not_incl = set(sig_genes) - set(scenic_meta_sel['Gene'])
genes_not_incl
```

Example 43 (unknown):
```unknown
{'AC008415.1',
 'AC009878.2',
 'AC011346.1',
 'AC018464.1',
 'AC022784.1',
 'AC063944.1',
 'AC097522.2',
 'AC105402.3',
 'AC119673.2',
 'AC124852.1',
 'AC137770.1',
 'ADAMTS9-AS2',
 'AL590004.3',
 'BANCR',
 'CCDC93',
 'CD81-AS1',
 'DANT2',
 'DHRSX',
 'DLEU2',
 'DUSP27',
 'ERO1A',
 'FSD2',
 'GAB2',
 'GALNTL6',
 'GPCPD1',
 'HAND2-AS1',
 'HIF1A-AS3',
 'JPX',
 'LINC00342',
 'LINC00881',
 'LINC00882',
 'LINC00906',
 'LINC01128',
 'LINC02248',
 'LINC02649',
 'LRRC52-AS1',
 'MALAT1',
 'MT-ATP6',
 'MT-ATP8',
 'MT-CO1',
 'MT-CO2',
 'MT-CO3',
 'MT-CYB',
 'MT-ND1',
 'MT-ND2',
 'MT-ND3',
 'MT-ND4',
 'MT-ND4L',
 'MT-ND5',
 'MT-ND6',
 'MTRNR2L1',
 'MTRNR2L12',
 'MTRNR2L8',
 'NEAT1',
 'NPIPB5',
 'PARM1',
 'PDE4DIP',
 'SLC16A1-AS1',
 'SLC16A9',
 'SMG1',
 'SNX30',
 'SPIRE1',
 'SV2C',
 'TRDN-AS1',
 'TRPM7',
 'TSIX',
 'TTN-AS1',
 'XIST',
 'ZNF780B'}
```

Example 44 (python):
```python
# add metrics
scenic_meta_sel['TF_n_linked_region'] = np.nan
scenic_meta_sel['TF_n_target_genes'] = np.nan
scenic_meta_sel['TF_n_target_sig_regions'] = np.nan

for tf in set(scenic_meta_sel['TF']):
    tf_ind = scenic_meta_sel[scenic_meta_sel['TF']==tf].index
    targets = list(set(scenic_meta_sel.loc[tf_ind,'Gene']))
    scenic_meta_sel.loc[tf_ind,'TF_n_target_genes'] = len(targets)
    regions = list(set(scenic_meta_sel.loc[tf_ind,'Region']))
    regions = list(set(regions).intersection(sig_peaks))
    scenic_meta_sel.loc[tf_ind,'TF_n_target_sig_regions'] = len(regions)
    if tf in list(set(scenic_meta_sel['Gene'])):
        gene_ind = scenic_meta_sel[scenic_meta_sel['Gene']==tf].index
        n_region = len(set(scenic_meta_sel.loc[gene_ind,'Region']))
        scenic_meta_sel.loc[tf_ind,'TF_n_linked_region'] = n_region
    else:
        scenic_meta_sel.loc[tf_ind,'TF_n_linked_region'] = 0
        
scenic_meta_sel.head(3)
```

Example 45 (unknown):
```unknown
scenic_meta_sel_TF_metrices = scenic_meta_sel[['TF','direct_TF','TF_n_target_genes','TF_n_target_sig_regions','TF_n_linked_region']]
# df = df[df['direct_TF']]
scenic_meta_sel_TF_metrices.drop_duplicates(inplace=True)
scenic_meta_sel_TF_metrices.set_index('TF',inplace=True)
scenic_meta_sel_TF_metrices.sort_values('TF_n_target_genes',ascending=False)[:30]
```

Example 46 (python):
```python
/tmp/ipykernel_12998/2002270316.py:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  scenic_meta_sel_TF_metrices.drop_duplicates(inplace=True)
```

Example 47 (python):
```python
# select interactions those targets are TFs
scenic_meta_sel_TF = scenic_meta_sel[scenic_meta_sel['Gene'].isin(list(set(scenic_meta_sel['TF'])))]

network_TF = {}
for tf in set(scenic_meta_sel_TF['TF']):
    df_sub = scenic_meta_sel_TF[scenic_meta_sel_TF['TF']==tf]
    network_TF[tf] = list(set(df_sub['Gene']))
print(len(network_TF.keys()))
```

Example 48 (unknown):
```unknown
centrality_type = 'eigenvector'
centrality_dict = regulon_plot.get_centrality(network_TF,
                   directed=True,
                   centrality_type=centrality_type, # or 'degree','in_dgree','out_degree'
                   out_df=False # if False output dictionary
                  )
centrality_dict
```

Example 49 (unknown):
```unknown
{'AHCTF1': 5.261440304720337e-13,
 'CUX1': 0.1613225717321188,
 'IRX3': 0.09655257585268091,
 'NFE2L1': 0.16804307262113158,
 'MEIS2': 0.17572204778120903,
 'GATA6': 0.258982861145849,
 'NFATC2': 0.058962883832198366,
 'HAND2': 0.16435825570985327,
 'NPAS2': 0.13296167973192152,
 'FOXK1': 0.09513524891186585,
 'ZBTB16': 0.19239868969576412,
 'PRDM16': 0.26446595865456146,
 'NCOA1': 0.09755874537895777,
 'PPARA': 0.25966245320131714,
 'ZNF148': 0.08260119221864297,
 'NFIC': 0.2550226201637516,
 'THRA': 0.20941008995417718,
 'NR4A1': 0.06297964019703133,
 'FOXN3': 0.052370866840318823,
 'KLF12': 8.6203437952538e-09,
 'SREBF2': 0.1669437114745006,
 'HMBOX1': 0.02829374150769981,
 'ESRRB': 8.6203437952538e-09,
 'PHF21A': 0.02829374150769981,
 'MEF2D': 8.6203437952538e-09,
 'ARID3A': 0.07904814500448452,
 'NFIB': 0.15573842049331146,
 'MEF2A': 0.15625062567673273,
 'GTF2IRD1': 0.17970394495998632,
 'BACH2': 8.6203437952538e-09,
 'ELF2': 0.07447411104479447,
 'ZNF586': 0.047890113112582074,
 'LCORL': 8.6203437952538e-09,
 'GATA4': 0.29973001663194976,
 'ATF1': 8.6203437952538e-09,
 'ZNF654': 0.02829374150769981,
 'TBX20': 0.27305250780466267,
 'ZSCAN18': 0.12383286193783795,
 'PBX1': 0.0217718774266034,
 'TFCP2': 0.1144017967214871,
 'MAX': 0.058524058266206304,
 'CREB1': 0.1174992030532011,
 'RORA': 0.058524058266206304,
 'TGIF2': 0.040827789573739046,
 'MXI1': 0.10796855806768811,
 'KLF6': 0.08828643755436749,
 'YY1': 0.07641069491594496,
 'HEY2': 0.20525881097301638,
 'USF2': 0.05999270466701148,
 'ZNF280D': 0.04311445515750737,
 'NR3C1': 0.20979980852423905,
 'ESRRA': 0.09800823377250537,
 'SOX5': 0.02829373288788216,
 'NFIA': 0.005123344739513594,
 'POU2F1': 5.261440304720337e-13,
 'BHLHE40': 0.04789010449276443,
 'ZBTB20': 0.03341708623932124,
 'MBNL2': 0.02829373288788216,
 'MAZ': 5.261440304720337e-13,
 'ZNF141': 5.261440304720337e-13,
 'SMAD3': 5.261440304720337e-13,
 'PLAG1': 0.04944449980200794,
 'FOXO1': 5.261440304720337e-13}
```

Example 50 (python):
```python
# add to dataframe and save
scenic_meta_sel_TF_metrices.loc[centrality_dict.keys(),f'centaliry_{centrality_type}'] = list(centrality_dict.values())
scenic_meta_sel_TF_metrices[f'centaliry_{centrality_type}'].fillna(0,inplace=True)
print(len(scenic_meta_sel_TF_metrices))
display(scenic_meta_sel_TF_metrices)

# save
# scenic_meta_sel_TF_metrices.to_csv('/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/TF_selection_df_compact-common-up_2Feb2024.csv')
```

Example 51 (unknown):
```unknown
set(marker_genes).intersection(sig_genes)
```

Example 52 (unknown):
```unknown
{'ATP2A2',
 'CACNA1C',
 'CASQ2',
 'CD36',
 'CKMT2',
 'COX6A2',
 'GJA1',
 'MLYCD',
 'PPARGC1A',
 'RYR2',
 'SCN5A'}
```

Example 53 (unknown):
```unknown
# TF-TG network
# utils.save_pkl(network,
#                '/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/TF-TG-network_compact-common-up_2Feb2024.pkl')

# TF-TF network
# utils.save_pkl(network_TF,
#                '/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/TF-TF-network_compact-common-up_2Feb2024.pkl')
```

Example 54 (python):
```python
import networkx as nx
```

Example 55 (unknown):
```unknown
# prepare parameters
size_scale = 7000

node_size_map = {}
for n in network_TF.keys():
    if n in centrality_dict.keys():
        node_size_map[n]=int(centrality_dict[n]*size_scale)
    else: # means no connection with other TFs
        node_size_map[n]=0

node_size_map
```

Example 56 (unknown):
```unknown
{'PPARA': 1817,
 'NFIB': 1090,
 'HEY2': 1436,
 'KLF12': 0,
 'PRDM16': 1851,
 'SREBF2': 1168,
 'ZNF141': 0,
 'CREB1': 822,
 'AHCTF1': 0,
 'CUX1': 1129,
 'GATA4': 2098,
 'TBX20': 1911,
 'LCORL': 0,
 'FOXO1': 0,
 'GATA6': 1812,
 'BHLHE40': 335,
 'ELF2': 521,
 'ZNF586': 335,
 'NFATC2': 412,
 'MAX': 409,
 'MEIS2': 1230,
 'MXI1': 755,
 'NR3C1': 1468,
 'MAZ': 0,
 'NPAS2': 930,
 'RORA': 409,
 'MEF2A': 1093,
 'BACH2': 0,
 'ZNF654': 198,
 'IRX3': 675,
 'NFIC': 1785,
 'HAND2': 1150,
 'SMAD3': 0,
 'MEF2D': 0,
 'GTF2IRD1': 1257,
 'NFE2L1': 1176,
 'KLF6': 618,
 'ESRRB': 0,
 'HMBOX1': 198,
 'POU2F1': 0,
 'SOX5': 198,
 'USF2': 419,
 'PHF21A': 198,
 'ZNF148': 578,
 'NFIA': 35,
 'YY1': 534}
```

Example 57 (unknown):
```unknown
regulon_plot.draw_networkx(
    network=network_TF,
    nodes_color_base='white',
    nodes_to_color=None, # list
    nodes_color_highlight=None,
    nodes_size_base=400,
    nodes_size_map=node_size_map,
     spring_layout_k=1.5,
    spring_layout_scale=1.5,
    figsize =(7,7),
    random_seed=12,
    numpy_random_seed=100
    
)
```

Example 58 (python):
```python
import networkx as nx

# Create a directed graph
G = nx.DiGraph()

# Add nodes and edges
for tf, targets in network.items():
    G.add_node(tf, type='TF', size=node_size_map.get(tf, 1))  # Add TF node
    for tg in targets:
        if tg not in G:
            G.add_node(tg, type='TG', size=500)  # Add TG node if not already added
        G.add_edge(tf, tg)  # Add edge

# Define tf_nodes
tf_nodes = [node for node, attr in G.nodes(data=True) if attr['type'] == 'TF']
tg_nodes = [node for node, attr in G.nodes(data=True) if attr['type'] == 'TG']
tg_nodes.sort()

# Identifying direct and indirect TFs
direct_tfs = set()
for tf, targets in network.items():
    for tg in targets:
        if tg in tg_nodes:
            direct_tfs.add(tf)

indirect_tfs = list(set(tf_nodes) - direct_tfs)  # TFs that are not in direct TFs
direct_tfs = list(direct_tfs)
indirect_tfs.sort()
direct_tfs.sort()

# Manually creating circular positions for indirect and direct TFs, and linear positions for TGs
def create_circular_positions(node_list, center_x, center_y, radius):
    angle_step = 2 * np.pi / len(node_list)
    return {node: (center_x + np.cos(i * angle_step) * radius, center_y + np.sin(i * angle_step) * radius) 
            for i, node in enumerate(node_list)}

# Create circular positions for indirect and direct TFs
direct_tf_pos = create_circular_positions(direct_tfs, 0.5, 0.5, 0.3)

# Linear positions for indirect TFs
indirect_tf_x = [0 for _ in indirect_tfs]  # All nodes have the same x-coordinate
indirect_tf_y = np.linspace(0.8, 0.2, len(indirect_tfs))  # Evenly spaced y-coordinates
indirect_tf_pos = {indirect_tf: (x, y) for indirect_tf, x, y in zip(indirect_tfs, indirect_tf_x, indirect_tf_y)}

# Linear positions for TGs
tg_x = [1 for _ in tg_nodes]  # All TG nodes have the same x-coordinate
tg_y = np.linspace(1, 0, len(tg_nodes))  # Evenly spaced y-coordinates
tg_pos = {tg: (x, y) for tg, x, y in zip(tg_nodes, tg_x, tg_y)}

# Combine positions
pos = {**indirect_tf_pos, **direct_tf_pos, **tg_pos}

# Update node colors: indirect TFs to "lightskyblue"
for node in G.nodes:
    if node in indirect_tfs:
        G.nodes[node]['color'] = 'mistyrose'
    else:
        G.nodes[node]['color'] = 'lightsalmon' if G.nodes[node]['type'] == 'TF' else 'white'

# node colors
node_colors = [data['color'] for node, data in G.nodes(data=True)]

# node edge colors
# change color for the ones highly expressed in foetal vs adult
node_edge_colors = ['red' if node in marker_genes else 'black' for node in G.nodes()]
node_edge_widths = [2 if node in marker_genes else 1 for node in G.nodes()]
# node_shapes = ['s' if node in tg_nodes else 'o' for node in G.nodes()]

# Plotting with updated positions
plt.figure(figsize=(10, 10))
nx.draw(G, pos, with_labels=True, node_color=node_colors, node_size=[G.nodes[node]['size'] for node in G.nodes()], 
        edge_color='grey', alpha=0.9, 
        edgecolors=node_edge_colors, linewidths=node_edge_widths, node_shape="o")
# plt.title("TF-TG Network (Three Levels: Indirect TFs, Direct TFs, TGs)")
plt.show()
```

Example 59 (unknown):
```unknown
l = ['CD36', 'OSBPL2', 'PITPNC1', 'SREBF2', 'SLC27A1']
[x for x in l if x in sig_genes]
```

Example 60 (unknown):
```unknown
['CD36', 'OSBPL2', 'PITPNC1', 'SREBF2']
```

Example 61 (unknown):
```unknown
df = scenic_meta_sel[['Region','TF']]
df.drop_duplicates(inplace=True)
df['TF_count'] = 1
df = df.groupby('Region').sum()
df.head()
```

Example 62 (python):
```python
ad = adata_peak[:,df.index]
ad = ad[ad.obs['fine_grain']==cell_of_intest]
ad
```

Example 63 (python):
```python
df[f'total_count, {cell_of_intest}'] = ad.to_df().sum().reindex(df.index)
plt.scatter(df['TF_count'],df[f'total_count, {cell_of_intest}'])
plt.xlabel('TF_count')
plt.ylabel(f'total_count, {cell_of_intest}')
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/TransmuralAxis/tm_axis_calculation_OrganAxis.html

**Contents:**
- Read modules¶
- Function¶
- List of samples¶
- Core axis calculation per slide¶

**Examples:**

Example 1 (unknown):
```unknown
TissueTag: OrganAxis
```

Example 2 (python):
```python
import os
import numpy as np 
# from bokeh.plotting import show, output_notebook
import pandas as pd
import socket
# os.chdir(path_to_tissue_tag)
# import tissue_tag as tt
import matplotlib.pyplot as plt
import scanpy as sc
# output_notebook()
# host = '5016'
```

Example 3 (python):
```python
# kk18
# based on a cell2location function
def select_multislide(adata, s, s_col='sample'):

    slides = adata[adata.obs[s_col].isin(s), :]
    s_keys = list(slides.uns['spatial'].keys())
    
    uns_dict = {}
    for ss in s:
        s_spatial=np.array(s_keys)[[ss in k for k in s_keys]][0]
        uns_dict[s_spatial]=slides.uns['spatial'][s_spatial]
    
    slides.uns['spatial'] = uns_dict

    return slides

# ny1, TissueTag:OrganAxis
def axis_2p_norm(
    Dist2ClusterAll,
    structure_list,
    weights = [1,1], 
):
    import warnings
    warnings.filterwarnings("ignore")
    # CMA calculations 
    fa = weights[0]
    fb = weights[0]
    axis = np.array([( fa*int(a) - fb*int(b) ) / ( fa*int(a) + fb*int(b) ) for a,b in zip(Dist2ClusterAll[structure_list[0]], Dist2ClusterAll[structure_list[1]])])
    return axis

# ny1, TissueTag:OrganAxis
def dist2cluster(df, annotation, ppm, KNN=4, calc_dist=True, logscale=False):
    """
    Calculates the minimal Euclidean distance of each point in space to an anatomical structure.

    Parameters
    ----------
    df : pandas.DataFrame
        Dataframe containing spatial data.
    annotation : str
        The column in df that contains the structure annotation.
    ppm : float
        the coordinates of df in pixel scale
    KNN : int, optional
        Determines the number of closest points used to calculate the mean distance. Default is 4.
    calc_dist : bool, optional
        If true, calculates the distance. Default is True.
    logscale : bool, optional
        If true, applies a log10 transformation to the distance. Default is False.
    

    Returns
    -------
    dict
        Dictionary with the median distance for each category in the annotation column.
    """
    pd.options.mode.chained_assignment = None  # default='warn'
    from scipy.spatial import distance_matrix, distance
    print('calculating distance matrix')
    a = np.vstack([df['x'],df['y']]).astype(int)
    distM = distance.cdist(a.T,a.T, metric='euclidean')

    Dist2ClusterAll = {}
    categories = np.unique(df[annotation])

    for idx, c in enumerate(categories):
        indextmp = df[annotation] == c
        if len(np.where(indextmp)[0]) > KNN:
            print(c)
            Dist2ClusterAll[c] = np.median(np.sort(distM[indextmp, :], axis=0)[range(KNN), :], axis=0)

    for c in categories: 
        if c != 'unassigned':
            if calc_dist:
                if logscale:
                    df["L2_dist_log10_"+annotation+'_'+c] = np.log10(Dist2ClusterAll[c]/ppm)
                else:
                    df["L2_dist_"+annotation+'_'+c] = Dist2ClusterAll[c]/ppm
            df[annotation] = categories[np.argmin(np.array(list(Dist2ClusterAll.values())), axis=0)]
    
    return Dist2ClusterAll
```

Example 4 (unknown):
```unknown
# selected visium slides which have transmural LV
# and annotated "tm_axis" (inmost or outmost) and "tm_axis_region-select" (LV)
path_to_annotation = '/nfs/team205/heart/visium_annotations/foetal/transmural'
samples = [x.split('_tm_axis.csv')[0] for x in os.listdir(path_to_annotation) if '_tm_axis.csv' in x]
samples.sort()
samples
```

Example 5 (unknown):
```unknown
['BHF_F_Hea13295830',
 'BHF_F_Hea13295831',
 'HCAHeartST10317184',
 'HCAHeartST10317185',
 'HCAHeartST10317186',
 'HCAHeartST11605165',
 'HCAHeartST11605167',
 'HCAHeartST11605168',
 'HCAHeartST13162330',
 'HCAHeartST13162331',
 'HCAHeartST13162336']
```

Example 6 (python):
```python
# sample metadata
meta=pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230208.csv')
meta=meta[['Publication','sangerID', 'combinedID',
       'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender',
       'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status']]
meta=meta[meta['modality'].isin(['Visium','Visium-FFPE'])]

# select slides for foetal publication
meta=meta[meta['Publication'].isin(['Foetal'])]

# samples for tm_axis
meta.set_index('sangerID',inplace=True)
meta = meta.loc[samples]
meta.reset_index(inplace=True)

# group
meta['group'] = meta['modality']+'_'+meta['status']+'_'+meta['region']
meta
```

Example 7 (unknown):
```unknown
# prepare dictionary
sample_dict = {}
for grp in set(meta['group']):
    sample_dict[grp] = list(meta[meta['group']==grp]['sangerID'])
sample_dict
```

Example 8 (unknown):
```unknown
{'Visium_Healthy_Left ventricle': ['HCAHeartST13162330', 'HCAHeartST13162331'],
 'Visium_Healthy_Heart': ['HCAHeartST10317184',
  'HCAHeartST10317185',
  'HCAHeartST10317186',
  'HCAHeartST11605165',
  'HCAHeartST11605167',
  'HCAHeartST11605168'],
 'Visium_Trisomy21_Heart': ['HCAHeartST13162336'],
 'Visium-FFPE_Healthy_Heart': ['BHF_F_Hea13295830', 'BHF_F_Hea13295831']}
```

Example 9 (unknown):
```unknown
!ls /nfs/team205/heart/anndata_objects/Foetal
```

Example 10 (python):
```python
14Feb_fromSemih		      RNA_foetal_post-scVI_raw.h5ad
archive			      RNA_foetal_raw.h5ad
Feb28ObjectRaw.h5ad	      scArches
Feb28ObjectRaw_Multiome.h5ad  scVI
Feb28ObjectRaw_Multiome.rds   TipAndStalkAndGVSMCsPartitionedForCell2Loc.h5ad
Feb28ObjectRaw.obs.csv	      trisomy21
Feb28ObjectRaw_original.h5ad  visium-FFPE_foetal_raw.h5ad
Feb28ObjectRaw_re-scVI.h5ad   visium_foetal_healthy-all_raw.h5ad
He_Cell_foetal-lung	      visium_foetal_healthy-all_spot-annotation.csv
Hill2022.h5ad		      visium_foetal_raw.h5ad
Hill2022_QCapplied.h5ad       visium-OCT_foetal_acrossage_raw.h5ad
Immune			      visium-OCT_foetal_DuctusA_raw.h5ad
metadata		      visium-OCT_foetal_raw.h5ad
multiome_ATAC		      visium-OCT-LV_foetal_raw.h5ad
multiVI			      visium-OCT-T21_foetal_raw.h5ad
QC
```

Example 11 (python):
```python
sc.set_figure_params(figsize=[5,5],dpi=100)
```

Example 12 (python):
```python
for grp in sample_dict.keys():
    print(f'##### {grp} #####')
    
    # read in visium adata object
    if grp=='Visium-FFPE_Healthy_Heart':
        adata_vis = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-FFPE_foetal_raw.h5ad')
    elif grp=='Visium_Healthy_Heart':
        adata_vis = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT_foetal_raw.h5ad')
    elif grp=='Visium_Healthy_Left ventricle':
        adata_vis = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT-LV_foetal_raw.h5ad')
    elif grp=='Visium_Trisomy21_Heart':
        adata_vis = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT-T21_foetal_raw.h5ad')    
    adata_vis.obs['array_row'] = adata_vis.obs['array_row'].astype('int')
    adata_vis.obs['array_col'] = adata_vis.obs['array_col'].astype('int')
    adata_vis.obsm['spatial'] = adata_vis.obsm['spatial'].astype('int')

    for slide_id in sample_dict[grp]:
        print(slide_id) 
        # select one slide
        slide = select_multislide(adata_vis, [slide_id], s_col='sample')

        # read in annotation file and add
        ## annotation to select
        annot_select_region = pd.read_csv(f'{path_to_annotation}/{slide_id}_tm_axis_select-region.csv',index_col=0)
        annot_select_region.index = slide_id+'_'+annot_select_region.index
        annot_select_region.fillna('none',inplace=True)
        slide.obs['tm_axis_select-region'] = annot_select_region['tm_axis_select-region'].reindex(slide.obs_names)
        ## annotation, transmural
        annot_tm = pd.read_csv(f'{path_to_annotation}/{slide_id}_tm_axis.csv',index_col=0)
        annot_tm.index = slide_id+'_'+annot_tm.index
        annot_tm.fillna('none',inplace=True)
        slide.obs['tm_annotation'] = annot_tm['tm_axis'].reindex(slide.obs_names)
        ## plot
        sc.pl.spatial(slide,color=['tm_axis_select-region','tm_annotation'],ncols=2,wspace=0.2)

        # prepare dataframe for distance calculation
        obs_df = slide.obs.copy()
        obs_df.rename(columns={'array_row':'y','array_col':'x'},inplace=True)

        regions_to_select = [x for x in set(obs_df['tm_axis_select-region']) if x!='none']
        # if the slide has one heart 
        if len(regions_to_select)==1: 
            # select region
            obs_df_sel = obs_df[obs_df['tm_axis_select-region']==regions_to_select[0]]
    
            # calculate distance
            print('calculating distances')
            dist2clusterall = dist2cluster(obs_df_sel,annotation='tm_annotation',KNN=1,ppm=1)
            slide.obs.loc[obs_df_sel.index,obs_df_sel.columns] = obs_df_sel.copy()
    
            # calculate core axis
            arr = axis_2p_norm(
                dist2clusterall,
                ['outmost','inmost'],
                weights = [1,1], 
            )
            slide.obs.loc[obs_df_sel.index,'transmural_axis'] = arr
        
        # if the slide has multiple heart
        elif len(regions_to_select)>1:
            for region in regions_to_select:
                # select region
                obs_df_sel = obs_df[obs_df['tm_axis_select-region']==region]
        
                # calculate distance
                print('calculating distances')
                dist2clusterall = dist2cluster(obs_df_sel,annotation='tm_annotation',KNN=1,ppm=1)
                slide.obs.loc[obs_df_sel.index,obs_df_sel.columns] = obs_df_sel.copy()
        
                # calculate core axis
                arr = axis_2p_norm(
                    dist2clusterall,
                    ['outmost','inmost'],
                    weights = [1,1], 
                )
                slide.obs.loc[obs_df_sel.index,'transmural_axis'] = arr

        # plot
        sc.pl.spatial(slide,color=['transmural_axis'],cmap='gist_rainbow',ncols=2,wspace=0.25)
        # save axis
        slide.obs[['transmural_axis']].to_csv(f'/nfs/team205/heart/visium_axis/foetal/transmural_LV/{slide_id}.csv')
```

Example 13 (unknown):
```unknown
##### Visium_Healthy_Left ventricle #####
HCAHeartST13162330
```

Example 14 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 15 (unknown):
```unknown
HCAHeartST13162331
```

Example 16 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 17 (unknown):
```unknown
##### Visium_Healthy_Heart #####
HCAHeartST10317184
```

Example 18 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 19 (unknown):
```unknown
HCAHeartST10317185
```

Example 20 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 21 (unknown):
```unknown
HCAHeartST10317186
```

Example 22 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 23 (unknown):
```unknown
HCAHeartST11605165
```

Example 24 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 25 (unknown):
```unknown
HCAHeartST11605167
```

Example 26 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 27 (unknown):
```unknown
HCAHeartST11605168
```

Example 28 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 29 (unknown):
```unknown
##### Visium_Trisomy21_Heart #####
HCAHeartST13162336
```

Example 30 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 31 (unknown):
```unknown
##### Visium-FFPE_Healthy_Heart #####
BHF_F_Hea13295830
```

Example 32 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 33 (unknown):
```unknown
BHF_F_Hea13295831
```

Example 34 (unknown):
```unknown
calculating distances
calculating distance matrix
inmost
none
outmost
```

Example 35 (unknown):
```unknown
!ls /nfs/team205/heart/visium_axis/foetal/transmural_LV
```

Example 36 (unknown):
```unknown
BHF_F_Hea13295830.csv	HCAHeartST10317186.csv	HCAHeartST13162330.csv
BHF_F_Hea13295831.csv	HCAHeartST11605165.csv	HCAHeartST13162331.csv
HCAHeartST10317184.csv	HCAHeartST11605167.csv	HCAHeartST13162336.csv
HCAHeartST10317185.csv	HCAHeartST11605168.csv
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/Visium_mapping/c2l_pt2_mappings_FFPE.html

**Contents:**
  - Loading packages¶
- Functions¶
- Read in metadata¶
- Data, parameters¶
- Loading visium data¶
- Filter out low quality spots¶
- Remove mitochondrial genes¶
- Add metadata and annotation¶
- Remove spots based on manual annotation¶
- Combine pcw13w-Atrial and pcw13w-Ventricle data to have whole heart in one sample¶
- Run per group: cell2location mapping¶

**Examples:**

Example 1 (unknown):
```unknown
run = "20Mar2024_fetal_FFPE"
```

Example 2 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 3 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Global seed set to 0
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 4 (python):
```python
import gc
import torch
```

Example 5 (python):
```python
def read_and_qc(sample_name, path):
    r""" This function reads the data for one 10X spatial experiment into the anndata object.
    It also calculates QC metrics. Modify this function if required by your workflow.

    :param sample_name: Name of the sample
    :param path: path to data
    """

    adata = sc.read_visium(path + str(sample_name),
                           count_file='filtered_feature_bc_matrix.h5', load_images=True)
    adata.var_names_make_unique(join='-')
    
    adata.obs['sample'] = sample_name
    
    # rename genes to ENSEMBL
    adata.var['SYMBOL'] = adata.var_names
    adata.var_names = adata.var['gene_ids']
    adata.var_names.name = None

    # Calculate QC metrics
    sc.pp.calculate_qc_metrics(adata, inplace=True)
    adata.var['mt'] = [gene.startswith('MT-') for gene in adata.var['SYMBOL']]
    adata.obs['mt_frac'] = adata[:, adata.var['mt'].tolist()].X.sum(1).A.squeeze()/adata.obs['total_counts']

    # add sample name to obs names
    adata.obs["sample"] = [str(i) for i in adata.obs['sample']]
    adata.obs_names = adata.obs["sample"] \
                          + '_' + adata.obs_names
    adata.obs.index.name = 'spot_id'

    return adata

def select_slide(adata, s, s_col='sample'):
    r""" This function selects the data for one slide from the spatial anndata object.

    :param adata: Anndata object with multiple spatial experiments
    :param s: name of selected experiment
    :param s_col: column in adata.obs listing experiment name for each location
    """

    slide = adata[adata.obs[s_col].isin([s]), :]
    s_keys = list(slide.uns['spatial'].keys())
    s_spatial = np.array(s_keys)[[s in k for k in s_keys]][0]

    slide.uns['spatial'] = {s_spatial: slide.uns['spatial'][s_spatial]}

    return slide
```

Example 6 (python):
```python
meta=pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230208_for-cell2loc.csv',
                index_col=0)
meta
```

Example 7 (python):
```python
pd.crosstab(meta['donor'],meta['cell2loc-map_group'])
```

Example 8 (unknown):
```unknown
# select slides
# Healthy, FFPE
meta = meta[meta['cell2loc-map_group'].isin(['FFPE_16PCW'])]

sampleIDs = list(meta['sangerID'])
len(sampleIDs)
```

Example 9 (python):
```python
import os
results_folder = os.path.join(os.getcwd(),"analysis/Visium/cell2location_results")

ref_run_name = f'{results_folder}/reference_signatures/{run}'
run_name = f'{results_folder}/cell2location_map/{run}'
```

Example 10 (unknown):
```unknown
visium_path='/nfs/team205/heart/mapped/spaceranger200/'

celltype_key='fine_grain_modified'
```

Example 11 (python):
```python
# Read the data of the interested region into anndata objects
slides = []
for i in sampleIDs:
    slides.append(read_and_qc(i, path=visium_path))

# Combine anndata objects together
adata_vis = slides[0].concatenate(
    slides[1:],
    batch_key="sample",
    uns_merge="unique",
    batch_categories=sampleIDs,
    index_unique=None
)
```

Example 12 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
```

Example 13 (python):
```python
# PLOT QC FOR EACH SAMPLE
fig, axs = plt.subplots(len(slides), 4, figsize=(15, 4*len(slides)-4))
for i, s in enumerate(adata_vis.obs['sample'].unique()):
    #fig.suptitle('Covariates for filtering')

    slide = select_slide(adata_vis, s)
    sns.distplot(slide.obs['total_counts'],
                 kde=False, ax = axs[i, 0])
    axs[i, 0].set_xlim(0, adata_vis.obs['total_counts'].max())
    axs[i, 0].set_xlabel(f'total_counts | {s}')

    sns.distplot(slide.obs['total_counts']\
                 [slide.obs['total_counts']<2000],
                 kde=False, bins=40, ax = axs[i, 1])
    axs[i, 1].set_xlim(0, 2000)
    axs[i, 1].set_xlabel(f'total_counts | {s}')

    sns.distplot(slide.obs['n_genes_by_counts'],
                 kde=False, bins=60, ax = axs[i, 2])
    axs[i, 2].set_xlim(0, adata_vis.obs['n_genes_by_counts'].max())
    axs[i, 2].set_xlabel(f'n_genes_by_counts | {s}')

    sns.distplot(slide.obs['n_genes_by_counts']\
                 [slide.obs['n_genes_by_counts']<1000],
                 kde=False, bins=60, ax = axs[i, 3])
    axs[i, 3].set_xlim(0, 1000)
    axs[i, 3].set_xlabel(f'n_genes_by_counts | {s}')

plt.tight_layout()
```

Example 14 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_1547584/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_1547584/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_1547584/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_1547584/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_1547584/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_1547584/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_1547584/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_1547584/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
```

Example 15 (python):
```python
# filter out low quality spots
sc.pp.filter_cells(adata_vis, min_counts = 500)
sc.pp.filter_cells(adata_vis, min_genes = 300)

# Spot number per slide
adata_vis.obs['sample'].value_counts()
```

Example 16 (unknown):
```unknown
BHF_F_Hea13295830    12280
BHF_F_Hea13295831     8899
Name: sample, dtype: int64
```

Example 17 (python):
```python
# find mitochondria-encoded (MT) genes
adata_vis.var['MT_gene'] = [gene.startswith('MT-') for gene in adata_vis.var['SYMBOL']]
print(sum(adata_vis.var['MT_gene']))
    
# remove MT genes for spatial mapping (keeping their counts in the object)
adata_vis.obsm['MT'] = adata_vis[:, adata_vis.var['MT_gene'].values].X.toarray()
adata_vis = adata_vis[:, ~adata_vis.var['MT_gene'].values]
print(sum(adata_vis.var['MT_gene']))
```

Example 18 (unknown):
```unknown
meta.head()
```

Example 19 (python):
```python
adata_vis.obs = adata_vis.obs.reset_index().merge(meta, how='left', left_on='sample', right_on='sangerID').set_index('spot_id')
adata_vis
```

Example 20 (unknown):
```unknown
AnnData object with n_obs × n_vars = 21179 × 18074
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'n_counts', 'n_genes', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group'
    var: 'gene_ids', 'feature_types', 'genome', 'SYMBOL', 'mt', 'n_cells_by_counts-BHF_F_Hea13295830', 'mean_counts-BHF_F_Hea13295830', 'log1p_mean_counts-BHF_F_Hea13295830', 'pct_dropout_by_counts-BHF_F_Hea13295830', 'total_counts-BHF_F_Hea13295830', 'log1p_total_counts-BHF_F_Hea13295830', 'n_cells_by_counts-BHF_F_Hea13295831', 'mean_counts-BHF_F_Hea13295831', 'log1p_mean_counts-BHF_F_Hea13295831', 'pct_dropout_by_counts-BHF_F_Hea13295831', 'total_counts-BHF_F_Hea13295831', 'log1p_total_counts-BHF_F_Hea13295831', 'MT_gene'
    uns: 'spatial'
    obsm: 'spatial', 'MT'
```

Example 21 (python):
```python
adata_vis.obs.head()
```

Example 22 (python):
```python
# annotation
path_to_annot='/nfs/team205/heart/visium_annotations/foetal'
category_name='JC_edit'

# read in annotation csv file and concatenate
for i, ID in enumerate(sampleIDs):
    if f'{ID}_JC.csv' in os.listdir(path_to_annot):
    
        print(ID)
        annot_df = pd.read_csv(f'{path_to_annot}/{ID}_JC.csv')
        annot_df['spot_id']=ID+'_'+annot_df['Barcode']

        if i==0:
            annot_df_all=annot_df.copy()
        else:
            annot_df_all=pd.concat([annot_df_all,annot_df])
    else:
        print(f'{ID} is not annotated')

# add annotations to adata_vis
adata_vis.obs=pd.concat([adata_vis.obs,
                     annot_df_all.set_index('spot_id')[[category_name]].reindex(adata_vis.obs.index)],
                     axis=1)
# replace nan to 'none'
adata_vis.obs[category_name].fillna('none', inplace=True)

# minor modification
annotation_edit_dict = {
     'AV Ring':'AV ring',
    'Aorta':'Great vessels',
    'Coronary Artery':'Coronary artery',
    'Mitral Valve':'AV valves',
    'Mitral valve':'AV valves',
    'Myocardium - Ventricular - Trabecular':'Myocardium - Ventricular - Trabeculated',
    'Myocardim - Ventricular - Trabeculated':'Myocardium - Ventricular - Trabeculated',
    'Myocardim - Ventricular - Compact':'Myocardium - Ventricular - Compact',
    'unknown':'Unannotated',
    'Unknown':'Unannotated',
}
adata_vis.obs.replace({category_name:annotation_edit_dict}, inplace=True)
set(adata_vis.obs[category_name])
```

Example 23 (unknown):
```unknown
BHF_F_Hea13295830
BHF_F_Hea13295831
```

Example 24 (unknown):
```unknown
{'AV ring',
 'AV valves',
 'AVN',
 'Aortic valve',
 'Bundle branch - left',
 'Bundle branch - right',
 'Cavity',
 'Connective tissue',
 'Coronary artery',
 'Epicardium',
 'Great vessels',
 'Great vessels ',
 'LN',
 'Myocardium - Atrial',
 'Myocardium - Papillary Muscle',
 'Myocardium - Ventricular - Compact',
 'Myocardium - Ventricular - Compact 1',
 'Myocardium - Ventricular - Trabeculated',
 'Valve apparatus'}
```

Example 25 (python):
```python
# remove 'none' and 'no_tissue'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['none','Out_of_tissue'])==False]

# remove 'Cavity'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['Cavity'])==False]

# remove 'Unannotated'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['Unannotated'])==False]
```

Example 26 (python):
```python
# order annotations
annotations_order = ['Myocardium - Atrial',
                     'Myocardium - Ventricular - Compact',
                     'Myocardium - Ventricular - Trabeculated',
                    'Myocardium - Papillary Muscle',
                    'Epicardium',
                     'AV valves',
                     'Aortic valve',
                    'Valve apparatus',
                   'AV ring',
                    'Connective tissue',
                     'Great vessels',
                     'Coronary artery',
                     # 'Nerve',
                     ]
#adata_vis.obs[category_name] = adata_vis.obs[category_name].astype('category')
#adata_vis.obs[category_name].cat.reorder_categories(annotations_order,inplace=True)
```

Example 27 (python):
```python
pd.crosstab(adata_vis.obs['cell2loc-map_group'],adata_vis.obs[category_name])
```

Example 28 (python):
```python
# check the slides for 'PCW-13w'
#mask = adata_vis.obs['age']=='PCW-13w'
#adata_vis[mask].obs['sample'].value_counts()
# 'HCAHeartST11605166': Atrium
# 'HCAHeartST11605165': Ventricle
```

Example 29 (python):
```python
# combine
adata_vis.obs['sample_for_cell2loc']=adata_vis.obs['sample'].astype('str')
#adata_vis.obs.replace({'sample_for_cell2loc':{
#    'HCAHeartST11605165':'HCAHeartST11605165_HCAHeartST11605166',
#    'HCAHeartST11605166':'HCAHeartST11605165_HCAHeartST11605166'
#}},inplace=True)
adata_vis.obs['sample_for_cell2loc'].value_counts()
```

Example 30 (python):
```python
/tmp/ipykernel_1547584/283590942.py:2: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
  adata_vis.obs['sample_for_cell2loc']=adata_vis.obs['sample'].astype('str')
```

Example 31 (unknown):
```unknown
BHF_F_Hea13295830    10848
BHF_F_Hea13295831     8899
Name: sample_for_cell2loc, dtype: int64
```

Example 32 (python):
```python
adata_vis.obs.columns
```

Example 33 (unknown):
```unknown
Index(['in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts',
       'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts',
       'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes',
       'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac',
       'n_counts', 'n_genes', 'sangerID', 'combinedID', 'donor', 'donor_type',
       'region', 'region_finest', 'age', 'gender', 'facility',
       'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status',
       'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc'],
      dtype='object')
```

Example 34 (unknown):
```unknown
set(meta['cell2loc-map_group'])
```

Example 35 (unknown):
```unknown
{'FFPE_16PCW'}
```

Example 36 (python):
```python
# convert adata_vis.var_names to gene_id
adata_vis.var.set_index('gene_ids',inplace=True)
adata_vis.var.head()
```

Example 37 (python):
```python
adata_vis.obs.head(3)
```

Example 38 (python):
```python
%%time
import cell2location
# read in reference signatures
inf_aver = pd.read_csv(f"{ref_run_name}/all_data/inf_aver.csv", index_col=0)

# find shared genes and subset both anndata and reference signatures
intersect = np.intersect1d(adata_vis.var_names, inf_aver.index)
adata_vis = adata_vis[:, intersect].copy()
inf_aver = inf_aver.loc[intersect, :].copy()
print(inf_aver.shape)

for group in set(meta['cell2loc-map_group']):
    print(f'####### {group} #######')
    
    # subset adata
    adata_vis_sub=adata_vis[adata_vis.obs['cell2loc-map_group']==group].copy()
    samples = list(set(adata_vis_sub.obs['sample_for_cell2loc']))
    samples.sort()
    print(f'samples in this run: {samples}')
    adata_vis_sub=adata_vis_sub.copy()
    # prepare anndata for cell2location model
    cell2location.models.Cell2location.setup_anndata(adata=adata_vis_sub, batch_key="sample_for_cell2loc")
    #scvi.data.setup_anndata(adata=adata_vis_sub, batch_key="sample_for_cell2loc")
    #scvi.data.view_anndata_setup(adata_vis_sub)

    # create and train the model
    mod = cell2location.models.Cell2location(
        adata_vis_sub, cell_state_df=inf_aver, 
        # the expected average cell abundance: tissue-dependent 
        # hyper-prior which can be estimated from paired histology:
        N_cells_per_location=20,
        # hyperparameter controlling normalisation of
        # within-experiment variation in RNA detection (using default here):
        detection_alpha=20
    ) 

    mod.train(max_epochs=20000, 
              # train using full data (batch_size=None)
              batch_size=None, 
              # use all data points in training because 
              # we need to estimate cell abundance at all locations
              train_size=1,
              use_gpu=True)
    mod.view_anndata_setup()
    # plot ELBO loss history during training, removing first 100 epochs from the plot
    mod.plot_history(1000)
    plt.legend(labels=['full data training']);

    # In this section, we export the estimated cell abundance (summary of the posterior distribution).
    adata_vis_sub = mod.export_posterior(
        adata_vis_sub, sample_kwargs={'num_samples': 1000, 'batch_size': mod.adata.n_obs, 'use_gpu': True}
    )

    # Save model
    mod.save(f"{run_name}/{group}", overwrite=True)

    # Save anndata object with results
    adata_vis_sub.write(f"{run_name}/{group}/sp.h5ad")

    # Examine reconstruction accuracy to assess if there are any issues with mapping
    # the plot should be roughly diagonal, strong deviations will signal problems
    mod.plot_QC()
    
    del mod,adata_vis_sub
    gc.collect()
    torch.cuda.empty_cache()
```

Example 39 (unknown):
```unknown
(13904, 64)
####### FFPE_16PCW #######
samples in this run: ['BHF_F_Hea13295830', 'BHF_F_Hea13295831']
```

Example 40 (unknown):
```unknown
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/configuration_validator.py:106: UserWarning: You passed in a `val_dataloader` but have no `validation_step`. Skipping val loop.
  rank_zero_warn("You passed in a `val_dataloader` but have no `validation_step`. Skipping val loop.")
You are using a CUDA device ('NVIDIA A100-SXM4-80GB') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py:1609: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=10). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
  rank_zero_warn(
```

Example 41 (unknown):
```unknown
Epoch 20000/20000: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [47:34<00:00,  6.62it/s, v_num=1, elbo_train=2.96e+8]
```

Example 42 (unknown):
```unknown
`Trainer.fit` stopped: `max_epochs=20000` reached.
```

Example 43 (unknown):
```unknown
Epoch 20000/20000: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [47:34<00:00,  7.01it/s, v_num=1, elbo_train=2.96e+8]
```

Example 44 (unknown):
```unknown
Anndata setup with scvi-tools version 0.20.3.
```

Example 45 (unknown):
```unknown
Setup via `Cell2location.setup_anndata` with arguments:
```

Example 46 (unknown):
```unknown
{
│   'layer': None,
│   'batch_key': 'sample_for_cell2loc',
│   'labels_key': None,
│   'categorical_covariate_keys': None,
│   'continuous_covariate_keys': None
}
```

Example 47 (unknown):
```unknown
Summary Statistics         
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃     Summary Stat Key     ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│         n_batch          │   2   │
│         n_cells          │ 19747 │
│ n_extra_categorical_covs │   0   │
│ n_extra_continuous_covs  │   0   │
│         n_labels         │   1   │
│          n_vars          │ 13904 │
└──────────────────────────┴───────┘
```

Example 48 (python):
```python
Data Registry                
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Registry Key ┃    scvi-tools Location    ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│      X       │          adata.X          │
│    batch     │ adata.obs['_scvi_batch']  │
│    ind_x     │   adata.obs['_indices']   │
│    labels    │ adata.obs['_scvi_labels'] │
└──────────────┴───────────────────────────┘
```

Example 49 (python):
```python
batch State Registry                             
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃         Source Location          ┃    Categories     ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['sample_for_cell2loc'] │ BHF_F_Hea13295830 │          0          │
│                                  │ BHF_F_Hea13295831 │          1          │
└──────────────────────────────────┴───────────────────┴─────────────────────┘
```

Example 50 (python):
```python
labels State Registry                      
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃      Source Location      ┃ Categories ┃ scvi-tools Encoding ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ adata.obs['_scvi_labels'] │     0      │          0          │
└───────────────────────────┴────────────┴─────────────────────┘
```

Example 51 (unknown):
```unknown
Sampling local variables, batch: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:35<00:00, 35.02s/it]
Sampling global variables, sample: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 999/999 [00:25<00:00, 39.42it/s]
```

Example 52 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/cell2location/models/base/_pyro_mixin.py:407: UserWarning: The figure layout has changed to tight
  plt.tight_layout()
```

Example 53 (unknown):
```unknown
CPU times: user 50min 37s, sys: 30.9 s, total: 51min 8s
Wall time: 51min 8s
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/g2g_L_v_RaCM_pseudotime_associated_genes.html

**Contents:**
- Tutorial on single-cell trajectory alignment using Genes2Genes¶
  - Min max normalize the pseudotime¶
  - Check the number of bins in the optimal binning structure using OptBinning package¶
  - Visualize the interpolation binning structure in terms of the cell type composition¶
  - Run G2G alignment¶
  - The alignment distribution across all genes¶
  - Run gene set overrepresentation analysis over the top k mismatching genes¶
- COMMON | HIGH SIMILARITY | INCREASING¶
- COMMON | HIGH SIMILARITY | DECREASING¶
- COMMON | HIGH SIMILALRITY | DECREASING¶
- LEFT | LOW SIMILALRITY | INCREASING¶
- Look for enrichment of CHD genes in any of these groups¶
  - Clustering genes using their alignments¶
  - Below is the Levenshtein distance heat map of all genes ordered based on the above clustering structure¶

G2G aims to guide downstream comparative analysis of single-cell reference and query systems along any axis of progression (e.g. differentiation pseudotime, disease/treatment response pseudotime etc.). This notebook describes how we can use G2G framework to infer and analyse gene-level trajectory alignments between a given reference and query dataset.

In this tutorial, we are going to compare trajectories between two treatment groups (PAM and LPS) of mouse bone marrow-derived dendritic cells from Shalek et al (2014). The single cell datasets and their pseudotime estimates were downloaded from https://github.com/shenorrLab/cellAlign (Alpert et al 2018) and packaged into adata objects. There are 2 gene modules: global (core antiviral module) and local (peaked inflammatory module) considered by Alpert et al (2018), and we use the local module as an example comparison.

Load the reference and query anndata objects

OptBinning estimates 14 optimal number of splits for both reference and query pseudotime distributions. Therefore we choose the same number of interpolation points.

For this dataset, we use the author-given time annotations (1h,2h,4h,6h) as the cell-type annotations.

Next we define a colormap of our choice for these annotations, and call the below function.

This is done by first creating an aligner object, passing and setting all relevant parameters. Next we align all gene pairs. (This step is parallelizing indepedenent gene-alignments to make the process time-efficient, however the computational time for an individual alignment will increase as the number of cells and/or the number of interpolation time points increase.

Now we can check the aggregate (average) alignment across all genes:

We can also visualize this alignment in terms of cell-type composition

We can also visualize an individual gene (e.g. JUNB), displaying its alignment statistics

To check only the cell plots of a gene alignment (e.g. SERTAD2)

The below attributes and functions can be used to examine any gene-alignment object

We can use the alignment similarity percentage statistic of genes to rank genes from highly distant to highly similar

1546 rows × 6 columns

1546 rows × 8 columns

1546 rows × 8 columns

Let us use 30% alignment similarity (=0.3) as a threshold in this case

1546 rows × 9 columns

We first run cluster diagnostics to decide on a distance threshold with a good tradeoff between the number of clusters and the quality of structure. We use levenshtein distance metric

We can inspect structures of distance thresholds that give local optimals of the mean silhouette score.

If we select distance threshold 0.37 which gives a local optimal of 0.3749 silhouette_score with 7 clusters

Below visualizes all alignment paths in each cluster along with its number of genes

1546 rows × 5 columns

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import gseapy as gp

def GO_plot(gene_list, gene_sets=['MSigDB_Hallmark_2020', 'KEGG_2021_Human', 'GO_Biological_Process_2023', 'Reactome_2022'], 
            display_table=False, title_suffix=None, sort_by='Combined Score', n_rank = 10, n_genes=None, FDR=None):
    """
    Generate GO plots and optionally display tables based on enrichment analysis.

    Parameters:
    - gene_list: List of gene names as strings
    - gene_sets: List of gene set names as strings
    - display_table: Boolean flag to display table or not
    - title_suffix: Suffix for the plot title
    - sort_by: Column name to sort the results and plot. Other options in table, e.g. 'n_genes'
    - n_rank: number of terms to include in plot
    """
    
    # Perform enrichment analysis
    enr_res = gp.enrichr(gene_list=gene_list,
                         gene_sets=gene_sets,
                         organism='human',
                         outdir=None)
    
    # Loop through each gene set to process and plot the results
    for geneset in gene_sets:
        print(geneset)
        
        # Filter the results based on the Gene_set
        df = enr_res.results[enr_res.results['Gene_set'] == geneset]
        
        # Calculate the number of genes in the overlap and add as a new column
        df['n_genes'] = [int(x.split('/')[0]) for x in df['Overlap']]
        
        # Sort the dataframe based on the specified column
        df = df.sort_values(sort_by, ascending=False)

        # Only use terms where a threshold number of genes are present
        if n_rank is not None:
            df = df[df.n_genes>=n_genes]

        if FDR is not None:
            df = df[df['Adjusted P-value']<FDR]
        
        # Optionally display the table
        if display_table:
            display(df[:30])
        
        # Generate the bar plot for the top 10 ranked terms
        plt.rcParams['axes.grid'] = False
        plt.rcParams['figure.figsize'] = 4, 3
        plt.barh(width=df[:n_rank][sort_by],
                 y=[x.split(' (')[0] for x in df[:n_rank]['Term']],
                 color='darkred')
        plt.gca().invert_yaxis()
        plt.xlabel(sort_by)
        if title_suffix:
            plt.title(f'{geneset}_{title_suffix}')
        else:
            plt.title(f'{geneset}')
        plt.margins(y=0.02)
        plt.show()
        
    return df
```

Example 2 (python):
```python
import anndata

def split_adata_by_modality(adata, modality_column, modality_values):
    """
    Splits an AnnData object into multiple AnnData objects based on a given modality column.
    
    Parameters:
        adata (anndata.AnnData): The original AnnData object.
        modality_column (str): The name of the column in adata.var that contains the modality information.
        modality_values (list): List of unique modalities that you want to split the adata into.
        
    Returns:
        dict: A dictionary where the keys are the modality names and the values are the new AnnData objects.
    """
    # Initialize an empty dictionary to hold the new AnnData objects
    splitted_adatas = {}
    
    # Loop through each modality value to create a new AnnData object
    for modality in modality_values:
        # Get the indices of the features belonging to the current modality
        feature_idx = adata.var[modality_column] == modality
        
        # Create a new AnnData object with only the features belonging to the current modality
        new_adata = anndata.AnnData(X=adata[:, feature_idx].X,
                                    obs=adata.obs,
                                    var=adata.var.loc[feature_idx],
                                    obsm=adata.obsm,
                                    uns=adata.uns)
        
        # Add the new AnnData object to the dictionary
        splitted_adatas[modality] = new_adata
    
    return splitted_adatas
```

Example 3 (python):
```python
import scanpy as sc
import anndata
import numpy as np
import seaborn as sb
import matplotlib.pyplot as plt
import platform
from optbinning import ContinuousOptimalBinning
import pandas as pd


from genes2genes import Main
from genes2genes import VisualUtils
from genes2genes import ClusterUtils
from genes2genes import TimeSeriesPreprocessor
from genes2genes import PathwayAnalyserV2

print(platform.python_version())
```

Example 4 (python):
```python
import matplotlib
matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42
```

Example 5 (unknown):
```unknown
temporal_ordering='age_days_smoothed_once'
```

Example 6 (python):
```python
paired_adata_with_time_path = "/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA/aCMs_WholeSample_paired_with_time.h5ad"
```

Example 7 (python):
```python
adata=sc.read(paired_adata_with_time_path)
adata
```

Example 8 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
    obsp: 'connectivities', 'distances'
```

Example 9 (python):
```python
adata.obs["time"]=adata.obs['t'].copy()
```

Example 10 (python):
```python
adata.X.data[:5]
```

Example 11 (unknown):
```unknown
array([2., 1., 1., 1., 1.], dtype=float32)
```

Example 12 (python):
```python
sc.pl.umap(adata,color=["seg","milestones","t","fine_grain"],ncols=2)
```

Example 13 (python):
```python
modality_column = "modality"
modality_values = ["Gene Expression", "Peaks"]
splitted_adatas = split_adata_by_modality(adata, modality_column, modality_values)

adata_gex = splitted_adatas["Gene Expression"]
print(adata_gex)

adata_atac = splitted_adatas["Peaks"]
print(adata_atac)
```

Example 14 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 36601
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones', 'time'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
AnnData object with n_obs × n_vars = 5664 × 508040
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones', 'time'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
```

Example 15 (python):
```python
sc.pl.umap(adata_gex,color='VASH2',cmap="RdPu")
```

Example 16 (python):
```python
# split intlo REF and QUERY
adata_ref=adata_gex[adata_gex.obs.fine_grain=="AtrialCardiomyocytesLeft"].copy()
adata_query=adata_gex[adata_gex.obs.fine_grain=="AtrialCardiomyocytesRight"].copy()
```

Example 17 (python):
```python
sc.pl.umap(adata_ref,color=["time","fine_grain"],ncols=2)
sc.pl.umap(adata_query,color=["time","fine_grain"],ncols=2)
```

Example 18 (python):
```python
sc.pp.normalize_total(adata_ref, target_sum=1e4)
sc.pp.log1p(adata_ref)

sc.pp.normalize_total(adata_query, target_sum=1e4)
sc.pp.log1p(adata_query)
```

Example 19 (python):
```python
print(adata_ref)
print(adata_query)
```

Example 20 (unknown):
```unknown
AnnData object with n_obs × n_vars = 3041 × 36601
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones', 'time'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors', 'log1p'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
AnnData object with n_obs × n_vars = 2623 × 36601
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones', 'time'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'donor_colors', 'fine_grain_colors', 'graph', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'seg_colors', 'umap', 'week_colors', 'log1p'
    obsm: 'X_R', 'X_multiVI', 'X_umap'
```

Example 21 (python):
```python
# check the current range
print(min(adata_ref.obs['time']), max(adata_ref.obs['time']))
print(min(adata_query.obs['time']), max(adata_query.obs['time']))

# if it does not follow [0,1] range, run below
adata_ref.obs['time'] = TimeSeriesPreprocessor.Utils.minmax_normalise(np.asarray(adata_ref.obs['time']))
adata_query.obs['time'] = TimeSeriesPreprocessor.Utils.minmax_normalise(np.asarray(adata_query.obs['time']))
```

Example 22 (unknown):
```unknown
0.4222375414474298 8.245970642037001
0.45019867983228623 8.303595486512195
```

Example 23 (python):
```python
# Visualize the pseudotime distributions
sb.kdeplot(adata_ref.obs['time'], fill=True, color='forestgreen') 
sb.kdeplot(adata_query.obs['time'], fill=True, color='midnightblue'); 
plt.xlabel('pseudotime'); plt.legend(); plt.show()
```

Example 24 (unknown):
```unknown
No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
```

Example 25 (python):
```python
x = np.asarray(adata_ref.obs.time)
optb = ContinuousOptimalBinning(name='pseudotime', dtype="numerical")
optb.fit(x, x)
print(len(optb.splits))

x = np.asarray(adata_query.obs.time)
optb = ContinuousOptimalBinning(name='pseudotime', dtype="numerical")
optb.fit(x, x)
print(len(optb.splits))
```

Example 26 (python):
```python
col = np.array(sb.color_palette('colorblind'))[range(2)]
col
```

Example 27 (unknown):
```unknown
array([[0.00392157, 0.45098039, 0.69803922],
       [0.87058824, 0.56078431, 0.01960784]])
```

Example 28 (python):
```python
joint_cmap={'AtrialCardiomyocytesLeft':col[0], 'AtrialCardiomyocytesRight':col[1]}
vs = VisualUtils.VisualUtils.get_celltype_composition_across_time(adata_ref, adata_query, n_points=14, 
                                     ANNOTATION_COLNAME='fine_grain', optimal_binning=False)
```

Example 29 (unknown):
```unknown
# trying max n points for optimal binning = 14
====================================================
Optimal equal number of bins for R and Q =  14
```

Example 30 (python):
```python
# reads in the gene lists defined in the other notebook

import csv
import os

# Directory containing the CSV files
input_folder = "gene_lists"

# Initialize an empty dictionary to store the results
gene_lists = {}

# Iterate over each file in the directory
for filename in os.listdir(input_folder):
    if filename.endswith(".csv"):
        key = filename[:-4]  # Remove the '.csv' extension to get the key
        with open(os.path.join(input_folder, filename), 'r') as file:
            reader = csv.reader(file)
            # Convert each row into an item of the list (assuming one item per row)
            gene_list = [row[0] for row in reader]
            gene_lists[key] = gene_list

for k, v in gene_lists.items():
    print(f"{k}: {len(v)}")
```

Example 31 (unknown):
```unknown
INCREASING_genes: 1178
LEFT_DECREASING_AUCgene: 4
LEFT_INCREASING_peaks: 0
DECREASING_AUCregion: 35
RIGHT_INCREASING_genes: 61
LEFT_INCREASING_AUCgene: 0
LEFT_DECREASING_genes: 97
LEFT_genes: 277
RIGHT_AUCregion: 0
LEFT_INCREASING_genes: 72
COMMON_AUCgene: 44
RIGHT_DECREASING_peaks: 0
LEFT_INCREASING_AUCregion: 0
LEFT_peaks: 0
COMMON_genes: 1334
RIGHT_DECREASING_AUCgene: 1
RIGHT_DECREASING_genes: 57
LEFT_AUCgene: 5
LEFT_DECREASING_peaks: 0
DECREASING_genes: 5174
RIGHT_AUCgene: 2
LEFT_AUCregion: 2
RIGHT_INCREASING_AUCregion: 0
INCREASING_AUCregion: 4
DECREASING_AUCgene: 35
COMMON_peaks: 40
RIGHT_DECREASING_AUCregion: 0
INCREASING_peaks: 1571
DECREASING_peaks: 39998
RIGHT_peaks: 1
RIGHT_genes: 206
COMMON_AUCregion: 24
RIGHT_INCREASING_AUCgene: 0
LEFT_DECREASING_AUCregion: 0
INCREASING_AUCgene: 4
RIGHT_INCREASING_peaks: 0
```

Example 32 (unknown):
```unknown
# Assuming LEFT_genes, RIGHT_genes, and COMMON_genes are already defined as lists
LEFT_genes_set = set(gene_lists['LEFT_genes'])
RIGHT_genes_set = set(gene_lists['RIGHT_genes'])
COMMON_genes_set = set(gene_lists['COMMON_genes'])

# Create the union of these sets
gene_list_set = LEFT_genes_set | RIGHT_genes_set | COMMON_genes_set
```

Example 33 (python):
```python
gene_list = list(gene_list_set)
print(len(gene_list))
```

Example 34 (python):
```python
# Gets a a list of all human TFs
#TFs = pd.read_csv("human_TF_list.csv", index_col=0, header=1)
#TFs = TFs.Name.unique().tolist()
#gene_list = [gene for gene in TFs if gene in adata_ref.var_names]
```

Example 35 (unknown):
```unknown
gene_list[:10]
```

Example 36 (unknown):
```unknown
['TEAD1',
 'KHDRBS2',
 'MYPN',
 'CTNND1',
 'AL008730.1',
 'HSPA6',
 'NPTN',
 'EXOC6B',
 'FREM2',
 'PPP4C']
```

Example 37 (python):
```python
%%time

import os
import pickle

# Path for the folder and file
folder_path = '/nfs/team205/jc48/projects/fetal/code/g2g_aligners'
file_path = os.path.join(folder_path, 'COMMON_LEFT_RIGHT_aligner.pkl')

# Create folder if it doesn't exist
os.makedirs(folder_path, exist_ok=True)

# Check if the file already exists
if os.path.isfile(file_path):
    # Read the aligner object from the file
    print("this specific aligner already exists, reading it in...")
    with open(file_path, 'rb') as file:
        aligner = pickle.load(file)
else:
    # If the file does not exist, create and configure the aligner
    print("aligner not already made, making it...")
    aligner = Main.RefQueryAligner(adata_ref, adata_query, gene_list, len(vs.optimal_bining_S))
    aligner.WEIGHT_BY_CELL_DENSITY = True
    aligner.WINDOW_SIZE = 0.1
    aligner.state_params = [0.99, 0.1, 0.7]
    aligner.optimal_binning = True
    aligner.opt_binning_S = vs.optimal_bining_S
    aligner.opt_binning_T = vs.optimal_bining_T

    # Align all pairs
    aligner.align_all_pairs()

    # Save the aligner object to the file
    with open(file_path, 'wb') as file:
        pickle.dump(aligner, file)
```

Example 38 (unknown):
```unknown
this specific aligner already exists, reading it in...
CPU times: user 2.83 s, sys: 706 ms, total: 3.54 s
Wall time: 3.54 s
```

Example 39 (unknown):
```unknown
aligner.get_aggregate_alignment()
```

Example 40 (unknown):
```unknown
Average Alignment:  IIIDMMDIDDMMMMMMMM
```

Example 41 (unknown):
```unknown
vs.visualize_gene_alignment("IDDDMMMMMMMMMIIIDID")#, cmap=joint_cmap)
```

Example 42 (unknown):
```unknown
VisualUtils.show_gene_alignment('PITX2', aligner, vs)#, joint_cmap)
```

Example 43 (unknown):
```unknown
IIIDIDIDIDIIIIIIIIDDDDDDDDDD
Optimal alignment cost: 72.073 nits
Alignment similarity percentage: 0.0 %
```

Example 44 (unknown):
```unknown
df = aligner.get_stat_df()
```

Example 45 (unknown):
```unknown
mean matched percentage: 
64.46 %
```

Example 46 (unknown):
```unknown
VisualUtils.plot_alignmentSim_vs_l2fc(df)
```

Example 47 (unknown):
```unknown
gene_list = df[(df["alignment_similarity_percentage"]<0.45)&(df["abs_l2fc"]>0)].sort_values('alignment_similarity_percentage', ascending=True).Gene.tolist()
gene_list[:10]
```

Example 48 (unknown):
```unknown
['PITX2',
 'VASH2',
 'PANCR',
 'NAV3',
 'NTM',
 'PCDH7',
 'ROR1',
 'DCBLD2',
 'KCNMB2-AS1',
 'ZNF385B']
```

Example 49 (unknown):
```unknown
# adjust 'alignment_similarity_percentage' so it is actuall a perecntage
df['alignment_similarity_percentage'] *= 100
df
```

Example 50 (python):
```python
LEFT_genes_set = set(gene_lists['LEFT_genes'])
RIGHT_genes_set = set(gene_lists['RIGHT_genes'])
COMMON_genes_set = set(gene_lists['COMMON_genes'])

def classify_trajectory(gene):
    if gene in LEFT_genes_set:
        return "LaCM"
    elif gene in RIGHT_genes_set:
        return "RaCM"
    else:
        return "Common"

# Add the 'trajectory' column
df['trajectory'] = df['Gene'].apply(classify_trajectory)





INCREASING_genes_set = set(gene_lists['INCREASING_genes'])
DECREASING_genes_set = set(gene_lists['DECREASING_genes'])

def classify_trend(gene):
    if gene in INCREASING_genes_set:
        return "Increasing"
    elif gene in RIGHT_genes_set:
        return "Decreasing"
    else:
        return "Unchanging"

# Add the 'trajectory' column
df['trend'] = df['Gene'].apply(classify_trend)

df
```

Example 51 (unknown):
```unknown
df[(df["opt_alignment_cost"]<50)&(df["abs_l2fc"]>4)].sort_values('alignment_similarity_percentage', ascending=True).head(20)
```

Example 52 (unknown):
```unknown
genes_of_interest = ["GATA4","PITX2","NTM"]
#genes_of_interest = ["ATP5F1B","NDUFA11","NDUFB11","NDUFAB1","ATP5MC2"]
```

Example 53 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(6, 6))

markers = {
    "Increasing":"^",
    "Decreasing":"v",
    "Unchanging":"o"
}

#y_axis = 'opt_alignment_cost'
y_axis = 'alignment_similarity_percentage'

sns.scatterplot(data=df, x='l2fc', y=y_axis, hue='trajectory', edgecolor="none",
                #style = 'trend', markers = markers,
                legend=False, alpha=0.8)

# Add a horizontal line at y=0.5
plt.axhline(50, color='gray', linestyle='--', linewidth = 0.5)

# Add text annotations for LaCM and RaCM
plt.text(0.1, 1.05, 'RaCM', transform=plt.gca().transAxes, ha='left', va='top', weight='semibold')
plt.text(0.9, 1.05, 'LaCM', transform=plt.gca().transAxes, ha='right', va='top', weight='semibold')

# Label specific genes
for gene in genes_of_interest:
    gene_data = df[df['Gene'] == gene]
    for _, row in gene_data.iterrows():
        plt.text(row['l2fc'], row[y_axis], gene, 
                 horizontalalignment='right', size='small', color='black', weight='regular', style='italic')

# Set labels
plt.xlabel("Log2 Fold Change")

if y_axis == "alignment_similarity_percentage":
    plt.ylabel("Alignment Similarity (%)")

if y_axis == "opt_alignment_cost":
    plt.ylabel("Alignment Cost (nits)")

# Save the plot
plt.savefig("similarity_v_logFC.pdf", format='pdf', bbox_inches='tight')

# Show the plot
plt.show()
```

Example 54 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

# Assuming 'df' is your DataFrame and 'genes_of_interest' is defined

plt.figure(figsize=(6, 6))

markers = {
    "Increasing": "^",
    "Decreasing": "v",
    "Unchanging": "o"
}

y_axis = 'alignment_similarity_percentage'  # or 'opt_alignment_cost' based on your need

# Plotting the main scatterplot
sns.scatterplot(data=df, x='l2fc', y=y_axis,
                #hue='trajectory',
                edgecolor="none",
                #style='trend', markers=markers,
                legend=False, alpha=0.8)

# Add a horizontal line at y=0.5
plt.axhline(50, color='gray', linestyle='--', linewidth=0.5)

# Add text annotations for LaCM and RaCM
plt.text(0.1, 1.05, 'RaCM', transform=plt.gca().transAxes, ha='left', va='top', weight='semibold')
plt.text(0.9, 1.05, 'LaCM', transform=plt.gca().transAxes, ha='right', va='top', weight='semibold')

# Label specific genes
for gene in genes_of_interest:
    gene_data = df[df['Gene'] == gene]
    for _, row in gene_data.iterrows():
        plt.text(row['l2fc'], row[y_axis], gene, 
                 horizontalalignment='right', size='medium', color='red', weight='regular', style='italic')

# Set labels
plt.xlabel("Log2 Fold Change")
if y_axis == "alignment_similarity_percentage":
    plt.ylabel("Alignment Similarity (%)")
elif y_axis == "opt_alignment_cost":
    plt.ylabel("Alignment Cost (nits)")

# Draw rectangles to indicate boundaries of zoom-in plots
from matplotlib.patches import Rectangle

# Adjusted rectangle for blue boundary (zoom-in area)
plt.gca().add_patch(Rectangle((0, 0), 5.5, 25, linewidth=1, linestyle='--', edgecolor='blue', facecolor='none', alpha=0.6))
# Adjusted rectangle for orange boundary (zoom-in area)
plt.gca().add_patch(Rectangle((-3, 0), 3, 25, linewidth=1, linestyle='--', edgecolor='orange', facecolor='none', alpha=0.6))
plt.savefig("./figures/similarity_v_DE.pdf", bbox_inches='tight')
# Show the main plot
plt.show()
```

Example 55 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

plt.figure(figsize=(5, 2.5))


# Filter the DataFrame for the first scatterplot
df_filtered_2 = df[(df['l2fc'] < 0) & (df['l2fc'] > -6) &
                   (df['alignment_similarity_percentage'] < 25) & 
                   (df['alignment_similarity_percentage'] >= 0)]

specific_genes = ['MIR100HG', 'PLCL1', 'DGKH']
y_adjustment = 2  # The y-axis adjustment

# Sort the DataFrame based on 'l2fc' to ensure consistent adjustment application
df_filtered_2_sorted = df_filtered_2.sort_values(by='l2fc')

# Apply y-axis adjustments for specific genes
for i, gene_name in enumerate(specific_genes):
    # Find the index of the gene in the DataFrame
    gene_idx = df_filtered_2_sorted[df_filtered_2_sorted['Gene'] == gene_name].index
    if len(gene_idx) > 0:
        # Adjust the 'alignment_similarity_percentage' by 0.05 units times the position in the specific_genes list
        df_filtered_2_sorted.loc[gene_idx, 'alignment_similarity_percentage'] += y_adjustment * (i + 1)



# Now, plot each point. Since the DataFrame is already adjusted, no need for further manipulation
for _, row in df_filtered_2_sorted.iterrows():
    plt.text(row['l2fc'], row['alignment_similarity_percentage'], row['Gene'],
             fontdict={'horizontalalignment': 'center', 'verticalalignment': 'center', 'size': 9, 'weight': 'regular', 'color': 'orange'},
             bbox={'boxstyle': "round4", 'fc': 'white', 'ec': 'orange'})

plt.xlabel("Log2 Fold Change")
plt.ylabel("Alignment Similarity (%)")
plt.xlim(-3, 0)
plt.ylim(0, max(df_filtered_2_sorted['alignment_similarity_percentage'].max() + 5, 25))
plt.savefig("./figures/low_similarity_RaCMs.pdf", bbox_inches='tight')
plt.show()
```

Example 56 (unknown):
```unknown
df_filtered_2_sorted
```

Example 57 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

plt.figure(figsize=(5, 2.5))

# Filter the DataFrame for the first scatterplot
df_filtered_1 = df[(df['l2fc'] > 0) & (df['l2fc'] < 6) &
                   (df['alignment_similarity_percentage'] < 25) & 
                   (df['alignment_similarity_percentage'] >= 0)]

specific_genes = ['PITX2', 'PANCR', 'PCDH7']
y_adjustment = 5  # The y-axis adjustment

# Sort the DataFrame based on 'l2fc' to ensure consistent adjustment application
df_filtered_1_sorted = df_filtered_1.sort_values(by='l2fc')

# Apply y-axis adjustments for specific genes
for i, gene_name in enumerate(specific_genes):
    # Find the index of the gene in the DataFrame
    gene_idx = df_filtered_1_sorted[df_filtered_1_sorted['Gene'] == gene_name].index
    if len(gene_idx) > 0:
        # Adjust the 'alignment_similarity_percentage' by 0.05 units times the position in the specific_genes list
        df_filtered_1_sorted.loc[gene_idx, 'alignment_similarity_percentage'] += y_adjustment * (i + 1)



# Now, plot each point. Since the DataFrame is already adjusted, no need for further manipulation
for _, row in df_filtered_1_sorted.iterrows():
    plt.text(row['l2fc'], row['alignment_similarity_percentage'], row['Gene'],
             fontdict={'horizontalalignment': 'center', 'verticalalignment': 'center', 'size': 9, 'weight': 'regular', 'color': 'blue'},
             bbox={'boxstyle': "round4", 'fc': 'white', 'ec': 'blue'})

plt.xlabel("Log2 Fold Change")
plt.ylabel("Alignment Similarity (%)")
plt.xlim(0, 5.5)
plt.ylim(0, max(df_filtered_1_sorted['alignment_similarity_percentage'].max() + 5, 25))
plt.savefig("./figures/low_similarity_LaCMs.pdf", bbox_inches='tight')
plt.show()
```

Example 58 (unknown):
```unknown
df_filtered_1_sorted
```

Example 59 (python):
```python
query_color = 'orange'
ref_color = 'blue'

def plotTimeSeries_cmap(gene, aligner, plot_cells = False, plot_mean_trend= False):
    
        al_obj = aligner.results_map[gene]
        plt.subplots(1,3,figsize=(15,3))
        plt.subplot(1,3,1)
        plotTimeSeriesAlignment(gene, aligner) 
        plt.subplot(1,3,2)
        max_val = np.max([np.max(np.asarray(aligner.ref_mat[al_obj.gene])), np.max(np.asarray(aligner.query_mat[al_obj.gene]))])
        min_val = np.min([np.min(np.asarray(aligner.ref_mat[al_obj.gene])), np.min(np.asarray(aligner.query_mat[al_obj.gene]))])
        g = sb.scatterplot(x=aligner.query_time, y=np.asarray(aligner.query_mat[al_obj.gene]), alpha=0.7, color = query_color, legend=False,linewidth=0.3, s=20)  
        plt.title('Query')
        plt.ylim([min_val-0.5,max_val+0.5])
        plt.subplot(1,3,3)
        g = sb.scatterplot(x=aligner.ref_time, y=np.asarray(aligner.ref_mat[al_obj.gene]), color = ref_color, alpha=0.7, legend=False,linewidth=0.3,s=20 ) 
        plt.title('Reference')
        plt.ylim([min_val-0.5,max_val+0.5])

def plotTimeSeriesAlignment_cmap(gene, aligner):  
    
        al_obj = aligner.results_map[gene]
        sb.scatterplot(x=al_obj.S.X, y=al_obj.S.Y, color = ref_color ,alpha=0.05, legend=False)#, label='Ref') 
        sb.scatterplot(x=al_obj.T.X, y=al_obj.T.Y, color = query_color ,alpha=0.05, legend=False)#, label ='Query') 
        plot_mean_trends(al_obj)
        plt.title(al_obj.gene)
        plt.xlabel('pseudotime')
        plt.ylabel('Gene expression')
        plt.axis('off')
        
        for i in range(al_obj.matched_region_DE_info.shape[0]):
            S_timebin = int(al_obj.matched_region_DE_info.iloc[i]['ref_bin'])
            T_timebin = int(al_obj.matched_region_DE_info.iloc[i]['query_bin']) 
            x_vals = [al_obj.matched_region_DE_info.iloc[i]['ref_pseudotime'],al_obj.matched_region_DE_info.iloc[i]['query_pseudotime']] 
            y_vals = [al_obj.S.mean_trend[S_timebin ], al_obj.T.mean_trend[T_timebin]] 
            plt.plot(x_vals, y_vals, color='black', linestyle='dashed', linewidth=1.5)

def plot_mean_trends(al_obj):
    # Plotting the mean trends for S and T with specified colors
    al_obj.S.plot_mean_trend(color=ref_color)
    al_obj.T.plot_mean_trend(color=query_color)
```

Example 60 (python):
```python
import matplotlib.pyplot as plt
from contextlib import redirect_stdout
import io


for gene in genes_of_interest:
    print(gene)
    # Redirect the output of the plot function to a StringIO object
    f = io.StringIO()
    with redirect_stdout(f):
        #VisualUtils.plotTimeSeries(gene, aligner, vs)
        #VisualUtils.plotTimeSeriesAlignment(gene, aligner)
        plotTimeSeriesAlignment_cmap(gene, aligner)

    
    # Now, the plot is created and can be saved
    plt.savefig(f"{gene}_time_series_alignment.png", format='png', bbox_inches='tight')
    plt.show()
```

Example 61 (unknown):
```unknown
df.loc[df['Gene']=='MB',:]
```

Example 62 (unknown):
```unknown
lowSimGenes = df[list(df['alignment_similarity_percentage'] < 30)]
topSimGenes = df[list(df['alignment_similarity_percentage'] > 30)]
```

Example 63 (unknown):
```unknown
highly_similar_common_genes = topSimGenes[topSimGenes['trajectory']=="Common"]['Gene'].tolist()
INCREASING_highly_similar_common_genes = [gene for gene in highly_similar_common_genes if gene in gene_lists['INCREASING_genes']]
len(INCREASING_highly_similar_common_genes)
```

Example 64 (unknown):
```unknown
gsea_results=GO_plot(gene_list = INCREASING_highly_similar_common_genes, n_genes=5, display_table=True,
        #gene_sets=['MSigDB_Hallmark_2020','GO_Biological_Process_2023','KEGG_2021_Human'],
        gene_sets=['MSigDB_Hallmark_2020'],
                    FDR=0.1)
```

Example 65 (unknown):
```unknown
MSigDB_Hallmark_2020
```

Example 66 (python):
```python
gsea_results["pct_genes_in_set"] = round((gsea_results["n_genes"] / len(INCREASING_highly_similar_common_genes)) * 100, 2)
gsea_results["neglog10"] = np.negative(np.log10(gsea_results['Adjusted P-value']))
gsea_results = gsea_results[gsea_results['Adjusted P-value']<0.05]
gsea_results
```

Example 67 (python):
```python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

# Assuming gsea_results is your DataFrame

# Sort the DataFrame in decreasing order of neglog10 values
gsea_results_sorted = gsea_results.sort_values(by="neglog10", ascending=True)

plt.figure(figsize=(2, 3))

# Iterate over each row in the sorted DataFrame and plot individually
for _, row in gsea_results_sorted.iterrows():
    edge_color = 'red' if row["neglog10"] > 2 else 'black'
    plt.scatter(x=row["neglog10"], y=row["Term"], edgecolor=edge_color, color='white', lw=1)

plt.title("COMMON | HIGH SIMILARITY | INCREASING")
plt.ylabel("MSigDB Term")
plt.xlabel("-log(FDR)")

# Save the figure
plt.savefig(fname="./figures/COMMON___HIGH_SIMILARITY___INCREASING.pdf", format='pdf', bbox_inches='tight')

# Show the plot
plt.show()
```

Example 68 (unknown):
```unknown
highly_similar_common_genes = topSimGenes[topSimGenes['trajectory']=="Common"]['Gene'].tolist()
DECREASING_highly_similar_common_genes = [gene for gene in highly_similar_common_genes if gene in gene_lists['DECREASING_genes']]
len(DECREASING_highly_similar_common_genes)
```

Example 69 (unknown):
```unknown
gsea_results=GO_plot(gene_list = DECREASING_highly_similar_common_genes, n_genes=5, display_table=True,
        #gene_sets=['MSigDB_Hallmark_2020','GO_Biological_Process_2023','KEGG_2021_Human'],
        gene_sets=['MSigDB_Hallmark_2020'],
                    FDR=0.1)
```

Example 70 (unknown):
```unknown
MSigDB_Hallmark_2020
```

Example 71 (python):
```python
gsea_results["pct_genes_in_set"] = round((gsea_results["n_genes"] / len(DECREASING_highly_similar_common_genes)) * 100, 2)
gsea_results["neglog10"] = np.negative(np.log10(gsea_results['Adjusted P-value']))
gsea_results = gsea_results[gsea_results['Adjusted P-value']<0.05]
gsea_results
```

Example 72 (python):
```python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

# Assuming gsea_results is your DataFrame

# Sort the DataFrame in decreasing order of neglog10 values
gsea_results_sorted = gsea_results.sort_values(by="neglog10", ascending=True)

plt.figure(figsize=(2, 3))

# Iterate over each row in the sorted DataFrame and plot individually
for _, row in gsea_results_sorted.iterrows():
    edge_color = 'red' if row["neglog10"] > 2 else 'black'
    plt.scatter(x=row["neglog10"], y=row["Term"], edgecolor=edge_color, color='white', lw=1)

plt.title("COMMON | HIGH SIMILARITY | DECREASING")
plt.ylabel("MSigDB Term")
plt.xlabel("-log(FDR)")

# Save the figure
plt.savefig(fname="./figures/COMMON___HIGH_SIMILARITY___DECREASING.pdf", format='pdf', bbox_inches='tight')

# Show the plot
plt.show()
```

Example 73 (unknown):
```unknown
terms = ["Fatty Acid Metabolism","Myogenesis"]

term_genes_dict = {}

for term in terms:
    genes_series = gsea_results[gsea_results["Term"] == term]['Genes'].str.split(";")
    term_associated_genes_list = genes_series.sum()
    term_genes_dict[term] = term_associated_genes_list
term_genes_dict
```

Example 74 (unknown):
```unknown
{'Fatty Acid Metabolism': ['HADHB', 'AUH', 'MLYCD', 'XIST', 'ACO2', 'PPARA'],
 'Myogenesis': ['CAMK2B',
  'DTNA',
  'COL4A2',
  'MB',
  'CDH13',
  'PDE4DIP',
  'SORBS1',
  'LARGE1',
  'PPFIA4',
  'SGCG']}
```

Example 75 (unknown):
```unknown
GENE_SUBSET = term_genes_dict["Myogenesis"]
aligner.get_aggregate_alignment_for_subset(GENE_SUBSET)
```

Example 76 (unknown):
```unknown
Average Alignment:  IIIDDMMMDMMMMMMMM
```

Example 77 (python):
```python
import matplotlib.pyplot as plt
from contextlib import redirect_stdout
import io


for gene in term_genes_dict["Myogenesis"]:
    print(gene)
    # Redirect the output of the plot function to a StringIO object
    f = io.StringIO()
    with redirect_stdout(f):
        #VisualUtils.plotTimeSeries(gene, aligner, vs)
        #VisualUtils.plotTimeSeriesAlignment(gene, aligner)
        plotTimeSeriesAlignment_cmap(gene, aligner)

    
    # Now, the plot is created and can be saved
    plt.savefig(f"{gene}_time_series_alignment.png", format='png', bbox_inches='tight')
    plt.show()
```

Example 78 (unknown):
```unknown
highly_similar_common_genes = topSimGenes[topSimGenes['trajectory']=="Common"]['Gene'].tolist()
DECREASING_highly_similar_common_genes = [gene for gene in highly_similar_common_genes if gene in gene_lists['DECREASING_genes']]
len(DECREASING_highly_similar_common_genes)
```

Example 79 (unknown):
```unknown
gsea_results=GO_plot(gene_list = DECREASING_highly_similar_common_genes, n_genes=5, display_table=False,
        #gene_sets=['MSigDB_Hallmark_2020','GO_Biological_Process_2023','KEGG_2021_Human'])
        gene_sets=['MSigDB_Hallmark_2020'],
                     FDR=0.1)
```

Example 80 (unknown):
```unknown
MSigDB_Hallmark_2020
```

Example 81 (python):
```python
gsea_results["pct_genes_in_set"] = round((gsea_results["n_genes"] / len(DECREASING_highly_similar_common_genes)) * 100, 2)
gsea_results["neglog10"] = np.negative(np.log10(gsea_results['Adjusted P-value']))
gsea_results = gsea_results[gsea_results['Adjusted P-value']<0.2]
gsea_results
```

Example 82 (python):
```python
plt.figure(figsize=(2,3))
sb.scatterplot(data = gsea_results, x = "neglog10", y = "Term", hue = "neglog10", legend = False, palette = "viridis", size = "pct_genes_in_set", edgecolor='black')
plt.title("COMMON | HIGH SIMILARITY | DECREASING")
plt.ylabel("MSigDB Term")
plt.xlabel("-log(adjusted p)")
plt.savefig(fname="./figures/COMMON___HIGH SIMILARITY___DECREASING.pdf", format='pdf', bbox_inches='tight')
plt.show()
```

Example 83 (unknown):
```unknown
lowly_similar_left_genes = lowSimGenes[lowSimGenes['trajectory']=="RaCM"]['Gene'].tolist()
INCREASING_lowly_similar_left_genes = [gene for gene in lowly_similar_left_genes if gene in gene_lists['INCREASING_genes']]
len(INCREASING_lowly_similar_left_genes)
```

Example 84 (unknown):
```unknown
gsea_results=GO_plot(gene_list = INCREASING_lowly_similar_left_genes, n_genes=1, display_table=True,
        gene_sets=['MSigDB_Hallmark_2020','GO_Biological_Process_2023','KEGG_2021_Human'])
        #gene_sets=['MSigDB_Hallmark_2020'],
                     #FDR=0.1)
```

Example 85 (unknown):
```unknown
MSigDB_Hallmark_2020
```

Example 86 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 87 (unknown):
```unknown
KEGG_2021_Human
```

Example 88 (python):
```python
gsea_results["pct_genes_in_set"] = round((gsea_results["n_genes"] / len(INCREASING_highly_similar_common_genes)) * 100, 2)
gsea_results["neglog10"] = np.negative(np.log10(gsea_results['Adjusted P-value']))
gsea_results = gsea_results[gsea_results['Adjusted P-value']<0.2]
gsea_results
```

Example 89 (python):
```python
plt.figure(figsize=(2,3))
sb.scatterplot(data = gsea_results, x = "neglog10", y = "Term", hue = "neglog10", legend = False, palette = "viridis", size = "pct_genes_in_set", edgecolor='black')
plt.title("COMMON | HIGH SIMILARITY | DECREASING")
plt.ylabel("MSigDB Term")
plt.xlabel("-log(adjusted p)")
plt.savefig(fname="./figures/COMMON___HIGH SIMILARITY___DECREASING.pdf", format='pdf', bbox_inches='tight')
plt.show()
```

Example 90 (python):
```python
import matplotlib.pyplot as plt
import seaborn as sb

# Assuming gsea_results is your DataFrame and it's already loaded.

# Create the figure.
plt.figure(figsize=(2, 3))

# Unique terms for plotting on y-axis.
terms = gsea_results['Term'].unique()

# Mapping terms to y-values.
term_to_y = {term: i for i, term in enumerate(terms)}

# Plot each point individually.
for _, row in gsea_results.iterrows():
    edge_color = 'red' if row['neglog10'] > 2 else 'black'
    plt.scatter(x=row['neglog10'], y=term_to_y[row['Term']], s=row['pct_genes_in_set']*10,  # Adjust size as needed
                color=sb.color_palette("viridis", as_cmap=True)(row['neglog10']), edgecolor=edge_color)

# Set the y-ticks to the term names.
plt.yticks(ticks=list(term_to_y.values()), labels=list(term_to_y.keys()))

# Set title and labels.
plt.title("COMMON | HIGH SIMILARITY | DECREASING")
plt.ylabel("MSigDB Term")
plt.xlabel("-log(adjusted p)")

# Remove the legend created by seaborn
plt.legend([],[], frameon=False)

# Save and show the plot.
plt.savefig(fname="./figures/COMMON___HIGH SIMILARITY___DECREASING.pdf", format='pdf', bbox_inches='tight')
plt.show()
```

Example 91 (unknown):
```unknown
terms = ["Fatty Acid Metabolism","Myogenesis", "Oxidative Phosphorylation"]

term_genes_dict = {}

for term in terms:
    genes_series = gsea_results[gsea_results["Term"] == term]['Genes'].str.split(";")
    term_associated_genes_list = genes_series.sum()
    term_genes_dict[term] = term_associated_genes_list
term_genes_dict
```

Example 92 (unknown):
```unknown
{'Fatty Acid Metabolism': ['HSP90AA1',
  'FH',
  'MDH2',
  'ECH1',
  'RAP1GDS1',
  'ENO3',
  'PRDX6',
  'YWHAH'],
 'Myogenesis': ['EIF4A2',
  'MEF2C',
  'TNNC1',
  'TPM2',
  'MYL3',
  'PGAM2',
  'HSPB2',
  'TNNI1',
  'ENO3'],
 'Oxidative Phosphorylation': ['FH',
  'SLC25A3',
  'NDUFB8',
  'NDUFB7',
  'COX4I1',
  'NDUFB3',
  'ATP5MC3',
  'UQCR11',
  'UQCR10',
  'ETFB',
  'MRPL34',
  'COX6A1',
  'COX5A',
  'PRDX3',
  'LDHB',
  'ATP5F1B',
  'UQCRFS1',
  'POLR2F',
  'CYC1',
  'ATP5MF',
  'COX8A',
  'TIMM8B',
  'ATP5PF',
  'ATP5PD',
  'NDUFA6',
  'GPX4',
  'ATP5PB',
  'MDH2',
  'ECH1',
  'IDH2',
  'ATP5F1D',
  'ATP5F1E',
  'OXA1L',
  'NDUFS6',
  'ATP5PO',
  'CYCS',
  'SLC25A5',
  'ATP6V0C',
  'SLC25A11',
  'SLC25A6']}
```

Example 93 (unknown):
```unknown
GENE_SUBSET = term_genes_dict["Oxidative Phosphorylation"]
aligner.get_aggregate_alignment_for_subset(GENE_SUBSET)
```

Example 94 (unknown):
```unknown
Average Alignment:  IIDMMMMIDDMMMMMMM
```

Example 95 (python):
```python
import matplotlib.pyplot as plt
from contextlib import redirect_stdout
import io


for gene in term_genes_dict["Oxidative Phosphorylation"]:
    print(gene)
    # Redirect the output of the plot function to a StringIO object
    f = io.StringIO()
    with redirect_stdout(f):
        #VisualUtils.plotTimeSeries(gene, aligner, vs)
        #VisualUtils.plotTimeSeriesAlignment(gene, aligner)
        plotTimeSeriesAlignment_cmap(gene, aligner)

    
    # Now, the plot is created and can be saved
    plt.savefig(f"{gene}_time_series_alignment.png", format='png', bbox_inches='tight')
    plt.show()
```

Example 96 (python):
```python
chd_genes = pd.read_excel("chdgene_table.xlsx", header=1, index_col=0)
chd_genes
```

Example 97 (unknown):
```unknown
ASD_genes = chd_genes.loc[chd_genes["CHD classification"].str.contains("ASD"),:].index.tolist()
ASD_genes
```

Example 98 (unknown):
```unknown
['ABL1',
 'ACTC1',
 'ACVR1',
 'ACVR2B',
 'ANKRD11',
 'ARID1A',
 'ARID1B',
 'B3GAT3',
 'BCOR',
 'BMPR2',
 'BRAF',
 'CDK13',
 'CHD4',
 'CHD7',
 'CHST14',
 'CITED2',
 'CREBBP',
 'DNAH11',
 'DNAH5',
 'EFTUD2',
 'ELN',
 'EVC',
 'EVC2',
 'FGFR2',
 'FLNA',
 'FOXC1',
 'FOXP1',
 'GATA4',
 'GATA5',
 'GATA6',
 'GLI3',
 'GPC3',
 'HAAO',
 'HDAC8',
 'HNRNPK',
 'HRAS',
 'KANSL1',
 'KAT6A',
 'KAT6B',
 'KDM2B',
 'KDM6A',
 'KMT2A',
 'KMT2D',
 'KRAS',
 'KYNU',
 'LTBP2',
 'LZTR1',
 'MAP2K1',
 'MAP2K2',
 'MED12',
 'MEIS2',
 'MYBPC3',
 'MYH6',
 'MYH7',
 'NF1',
 'NIPBL',
 'NKX2-5',
 'NODAL',
 'NONO',
 'NOTCH2',
 'NPHP3',
 'NR2F2',
 'NSD1',
 'PBX1',
 'PIGV',
 'PITX2',
 'PTPN11',
 'RAB23',
 'RAF1',
 'RERE',
 'RIT1',
 'SALL1',
 'SETD5',
 'SHOC2',
 'SMAD2',
 'SMAD4',
 'SMARCA4',
 'SMARCB1',
 'SMARCE1',
 'SMC1A',
 'SMC3',
 'SON',
 1,
 'STRA6',
 'TAB2',
 'TBX20',
 'TBX5',
 'TGFBR1',
 'TGFBR2',
 'THOC6',
 'TLL1',
 'TMEM260',
 'TRAF7',
 'TXNL4A',
 'UBR1',
 'WASHC5',
 'ZEB2',
 'ZIC3']
```

Example 99 (unknown):
```unknown
all_CHD_genes = chd_genes.index.tolist()
```

Example 100 (unknown):
```unknown
df['CHD_associated'] = False
df.loc[df['Gene'].isin(all_CHD_genes),'CHD_associated'] = True
df
```

Example 101 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(6, 6))

markers = {
    "Increasing":"^",
    "Decreasing":"v",
    "Unchanging":"o"
}

#y_axis = 'opt_alignment_cost'
y_axis = 'alignment_similarity_percentage'

sns.scatterplot(data=df, x='l2fc', y=y_axis, hue='CHD_associated', edgecolor="none",
                style = 'trend', markers = markers,
                legend=False, alpha=0.8)

# Add a horizontal line at y=0.5
plt.axhline(0.5, color='gray', linestyle='--', linewidth = 0.5)

# Add text annotations for LaCM and RaCM
plt.text(0.1, 1.05, 'RaCM', transform=plt.gca().transAxes, ha='left', va='top', weight='semibold')
plt.text(0.9, 1.05, 'LaCM', transform=plt.gca().transAxes, ha='right', va='top', weight='semibold')

# Label specific genes
for gene in genes_of_interest:
    gene_data = df[df['Gene'] == gene]
    for _, row in gene_data.iterrows():
        plt.text(row['l2fc'], row[y_axis], gene, 
                 horizontalalignment='right', size='small', color='black', weight='regular', style='italic')

# Set labels
plt.xlabel("Log2 Fold Change")

if y_axis == "alignment_similarity_percentage":
    plt.ylabel("Alignment Similarity (%)")

if y_axis == "opt_alignment_cost":
    plt.ylabel("Alignment Cost (nits)")

# Save the plot
#plt.savefig("similarity_v_logFC.pdf", format='pdf', bbox_inches='tight')

# Show the plot
plt.show()
```

Example 102 (python):
```python
#sns.stripplot(data = df, x = 'CHD_associated', y = 'alignment_similarity_percentage', hue = 'CHD_associated', legend=False, jitter=0.05, alpha = 0.1)
sns.violinplot(data = df, x = 'trajectory', y = 'alignment_similarity_percentage', hue = 'CHD_associated', legend=False, alpha = 0.2, bw_adjust=2)
```

Example 103 (unknown):
```unknown
<Axes: xlabel='trajectory', ylabel='alignment_similarity_percentage'>
```

Example 104 (python):
```python
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats

# Your DataFrame manipulation and t-test code here
df['alignment_similarity_percentage'] = pd.to_numeric(df['alignment_similarity_percentage'], errors='coerce')

# Define the trajectories to be tested
trajectories = ['Common', 'LaCM', 'RaCM']
p_values = {}

# Perform t-tests and store p-values in a dictionary with trajectories as keys
for trajectory in trajectories:
    group1 = df[(df['trajectory'] == trajectory) & (df['CHD_associated'] == True)]['alignment_similarity_percentage'].dropna()
    group2 = df[(df['trajectory'] == trajectory) & (df['CHD_associated'] == False)]['alignment_similarity_percentage'].dropna()
    t_stat, p_value = stats.ttest_ind(group1, group2)
    p_values[trajectory] = p_value

# Plotting the violin plot
plt.figure(figsize=(10,6))
#ax = sns.violinplot(data=df, x='trajectory', y='alignment_similarity_percentage', hue='CHD_associated', alpha=0.2, bw_adjust=2, order=trajectories)
ax = sns.boxplot(data=df, x='trajectory', y='alignment_similarity_percentage', hue='CHD_associated', order=trajectories)

# Remove top and right spines
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)

# Function to add significance text
def add_significance_text(x_pos, y, text, color='k'):
    plt.text(x_pos, y, text, ha='center', va='bottom', color=color)

# Adding the statistical annotations
y_max = df['alignment_similarity_percentage'].max()
h = 2  # Offset for the annotation

# Annotate each group with its corresponding p-value
for i, trajectory in enumerate(trajectories):
    p_value = p_values[trajectory]
    p_text = f"p={p_value:.2e}" if p_value < 0.001 else f"p={p_value:.3f}"
    add_significance_text(i, y_max + h, p_text)


plt.show()
```

Example 105 (unknown):
```unknown
GO_plot(gene_list = topDEgenes[topDEgenes['trajectory']=="LaCM"]['Gene'].tolist(), n_genes=3, display_table=True)
```

Example 106 (unknown):
```unknown
MSigDB_Hallmark_2020
```

Example 107 (unknown):
```unknown
KEGG_2021_Human
```

Example 108 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 109 (unknown):
```unknown
Reactome_2022
```

Example 110 (unknown):
```unknown
# Running experiment to determine the distance threshold with a good trade-off
df = ClusterUtils.run_clustering(aligner, metric='levenshtein', experiment_mode=True)
```

Example 111 (unknown):
```unknown
compute distance matrix
using levenshtein distance metric
```

Example 112 (unknown):
```unknown
78%|██████████████████████████████████████████████████████████████████████████████████▍                       | 77/99 [00:22<00:06,  3.39it/s]
```

Example 113 (unknown):
```unknown
df.sort_values('Distance threshold', ascending=False)
```

Example 114 (unknown):
```unknown
aligner.show_ordered_alignments()
```

Example 115 (unknown):
```unknown
Gene          Alignment
------------  ----------------------------
PITX2         IIIDIDIDIDIIIIIIIIDDDDDDDDDD
DCBLD2        IIIDIDIDIDIIIIIIIIDDDDDDDDDD
PANCR         IIIDIDIDIDIIIIIIIIDDDDDDDDDD
PCDH7         IIIDIDIDIDIIIIIIIIDDDDDDDDDD
NAV3          IIIDIDIDIDIIIIIIIIDDDDDDDDDD
VASH2         IIIDIDIDIDIIIIIIIIDDDDDDDDDD
NTM           IIIDIDIDIDIIIIIIIIDDDDDDDDDD
ROR1          IIIDIDIDIDIIIIIIIIDDDDDDDDDD
GLCCI1        MMMMMMMMMMMMMM
MORF4L1       MMMMMMMMMMMMMM
PJA2          MMMMMMMMMMMMMM
RO60          MMMMMMMMMMMMMM
NAP1L1        MMMMMMMMMMMMMM
PPP4C         MMMMMMMMMMMMMM
CWC27         MMMMMMMMMMMMMM
MAPKAPK2      MMMMMMMMMMMMMM
MARCH6        MMMMMMMMMMMMMM
AL008730.1    MMMMMMMMMMMVVVDDD
TRIM24        MMMMMMMMMMMMMM
CITED2        MMMMMMMMMMMMMM
CT69          MMMMMMMMMMMMMM
ERBIN         MMMMMMMMMMMMMM
NIBAN1        MMMMMMMMMMMMMM
GATA4         MMMMMMMMMMMMMM
LDHA          MMMMMMMMMMMMMM
GAB1          MMMMMMMMMMMMMM
IDI2-AS1      MMMMMMMMMMMMMM
EXT1          MMMMMMMMMMMMMM
CCDC144A      MMMMMMMMMMMMMM
RPS5          MMMMMMMMMMMMMM
WDR37         MMMMMMMMMMMMMM
YWHAE         MMMMMMMMMMMMMM
FOXJ3         MMMMMMMMMMMMMM
AFF4          MMMMMMMMMMMMMM
PITPNM2       MMMMMMMMMMMMMM
SEPTIN7       MMMMMMMMMMMMMM
CNN1          MMMMMMMMMMMMMM
LIMS2         MMMMMMMMMMMMMM
ANKRD12       MMMMMMMMMMMMMM
ZNF106        MMMMMMMMMMMMMM
LINC01608     MMMMMMMVVVVVVVDDDDDDD
UBE4B         MMMMMMMMMMMMMM
USP24         MMMMMMMMMMMMMM
CSNK1A1       MMMMMMMMMMMMMM
LINC01913     MMMMMMMMMVVVVVDDDDD
AC002310.1    MMMMMMMMVVVVVVDDDDDD
XRN1          MMMMMMMMMMMMMM
ZDHHC21       MMMMMMMMMMMMMM
CD81          MMMMMMMMMMMMMM
FABP2         MMMMMMMMMWWWWWIIIII
UBR5          MMMMMMMMMMMMMM
DMAC1         MMMMMMMMMMMMMM
KANK2         MMMMMMMMMMMMMM
ZNF780B       MMMMMMMMMMMMMM
TPT1-AS1      MMMMMMMMMMMMMM
XRCC5         MMMMMMMMMMMMMM
FSTL1         MMMMMMMMMMMMMM
SRP9          MMMMMMMMMMMMMM
SNHG32        MMMMMMMMMMMMMM
LEPROT        MMMMMMMMMMMMMM
MGAT4C        MMMMMMMMMMMMMM
ABCC5         MMMMMMMMMMMMMM
NNT-AS1       MMMMMMMMMMMMMM
ACKR3         MMMMMMMMMWWWIIIMM
BMP10         MMMMWWWWWDDMMIIDIIIIII
LGALSL-DT     MMMMMMMMMIDMMMM
ARHGEF26-AS1  MMMMMMMMMIIDDMMM
TOP1MT        MMMMMMMMMIDMMMM
AC046158.1    MMMMMMMMMIDMMMM
PRX           MMMMMMMMMIDMMMM
HSF2          MMVVVVVVDDDDDDMMMMMM
MTRNR2L1      MMMMMMMMIDMMMMM
KCNK1         MMMMMMMMIIMMMDIDD
MARK2         MMVVVVVVDDDDDDMMMMMM
INTS5         MMMMMMMMIDMMMMM
PRDM5         MMVVVVVDDDIDDDMMMMMM
CALD1         MVVVVVVDDDDDDMMMMMMM
KIAA1324L     MMVVVVVDDDDDMMMMMMM
AL009177.1    MMMMWWWIIIMMMMMMM
ESRRB         MMMVVVVDDDDMMMMMMM
ST3GAL1       MMVVVVVDDDDDDDMMMMIIID
SP100         MMMMMMMIDMMMMMM
ALG10B        MMMMMMMIIDDMMMMM
AL137003.2    MMVVVVVDDDDDMMMMMMM
PKIG          MMWWWWWIIIIIMMMMMMM
AL160191.1    MMMMMMMIDIDMMMMM
STT3B         MMVVVVDIDDDDMMMMMMM
AKR1B1        MVVVVVDIDIDDDDDMMMMMM
YWHAH         MMVVVVDDDIDDMMMMMMM
HBG1          MMVVVVDDDDMMMMMMMM
FH            MMMMMMIDMMMMMMM
PPA1          MVVVVVDIDIDDDDDMMMMMM
TARBP1        MMVVVVDDDDMMMMMMMM
PSMB4         MMMMMIDIDMMMMMMM
HMGN2         MMMMMIDIDMMMMMMM
ETFB          MMMMMIDIDMMMMMMM
FILIP1        MMMMMIDMMMMMMMM
DHX32         MMMMMIDMMMMMMMM
SLC25A11      MMMMMIDIDMMMMMMM
PTTG1IP       MMMMMIDMMMMMMMM
ZNHIT1        MMMMMIDIDMMMMMMM
PEBP1         MMMMMIDIDMMMMMMM
NDUFB8        MMMMMIDIDMMMMMMM
AOPEP         MMMMMIDMMMMMMMM
CDC42         MMMMMIIDDMMMMMMM
MZT2B         MMMMMIDMMMMMMMM
ZFP36L1       MMMMMIIDDMMMMMMM
H2AFV         MMMMMIDIDMMMMMMM
VPS8          MMMMMIDIDIDMMMMMM
GPAT3         MMMMMIDIDMMMMMMM
EIF4A2        MMMMMIIDDMMMMMMM
GTF2IRD1      MMMMMIDIDMMMMMMM
EIF1          MMMMMIDIDMMMMMMM
HIGD2A        MMMMMIDIDMMMMMMM
CCSER2        MMMMMIDMMMMMMMM
POMP          MMMMMIDIDMMMMMMM
ATP5PO        MMMMMIDIDMMMMMMM
DUSP16        MMVVVDDDMMMMMMMMM
RACK1         MMMMMIIDDMMMMMMM
UGGT2         MMMMMIDMMMMMMMM
CASZ1         MMMMMIDMMMMMMMM
ANKH          MMMMMIDMMMMMMMM
ADI1          MMMMMIDMMMMMMMM
NDUFB2        MMMMMIIDDMMMMMMM
DYNLT1        MMMMMIDMMMMMMMM
OAZ1          MMMMMIIDDIDMMMMMM
SYNE1         MMMMMIDIDMMMMMMM
MT-ND2        MMMMMIDMMMMMMMM
TUT4          MMMMMIDMMMMMMMM
MARC2         MMMMMIDMMMMMMMM
PSPC1         MMMMMIDIDMMMMMMM
PRDX5         MMMMMIDIDMMMMMMM
HSPA6         MMMMIDMMMMMMMMM
KDM4B         MMMMIDMMMMMMMMM
KCNN2         MMMMDDDMMMMMMMIII
ATP5MC1       MMMMIDIDIDIDMMMMMM
KDELR1        MMMMDDMMMMMMMMVV
HBZ           MMMMIIIIIMMMMMWWWWW
COP1          MMMMIIDDMMMMMMMM
ZNF91         MMMMDDMMMMMMMMVV
RPS14         MMMMIDMMMMMMMMM
ST7           MMMMIDIDIDMMMMMMM
AL136171.2    MMMMIDMMMMMMMMM
BBX           MMMMIDMMMMMMMMM
BSG           MMMMIIDDMMMMMMMM
YWHAQ         MMMMIDMMMMMMMMM
USH1C         MMMMDDDMMMMMMMIII
MTUS2         MMMMIDMMMMMMMMM
ALS2          MMMMIDIDMMMMMMMM
MYO18A        MMMMIDMMMMMMMMM
HSPD1         MMMMIIDDIDMMMMMMM
COX4I1        MMMMIIDDIDMMMMMMM
NRP1          MMMMIIDIDDMMMMMMM
HSF1          MMMMIDIDIDMMMMMMM
PPP2R2D       MMMMIDMMMMMMMMM
EIF3K         MMMMIIDDIDIDMMMMMM
EIF4G2        MMMMDDMMMMMMMMVV
PARD3B        MMMMIDMMMMMMMMM
RAN           MMMMDDMMMMMMMMVV
SUN1          MMMMDDMMMMMMMMVV
ATP5F1D       MMMMIDIDIDMMMMMMM
EIF3E         MMMMDDMMMMMMMMVV
PPARA         MMMMIDIDMMMMMMMM
PDZRN3        MMMMIDIDMMMMMMMM
C17orf49      MMMMDDMMMMMMMMVV
FNIP1         MMMMDDMMMMMMMMVV
BMP5          MMMMIDIDMMMMMMMM
NDUFB4        MMMMIDIDIDMMMMMMM
FXR1          MMMMIDIDMMMMMMMM
TRMT9B        MMMMIDIDMMMMMMMM
PSMB2         MMMMIIDIDDMMMMMMM
NUCKS1        MMMMIDMMMMMMMMM
CUX1          MMMMIDIDMMMMMMMM
PRDX3         MMMMDDMMMMMMMMVV
ATP5F1C       MMMMIDIDIDMMMMMMM
SELENOW       MMMMDDMMMMMMMMVV
RPS27A        MMMMDDMMMMMMMMII
NEAT1         MMMMDDMMMMMMIIMM
COPA          MMMMIIDDMMMMMMMM
TOM1L2        MMMMIDIDMMMMMMMM
SPAG7         MMMMIIDDIDIDMMMMMM
ECH1          MMMMIDMMMMMMMMM
CISD1         MMMMIDIDIDMMMMMMM
GPC6          MMMMIDIDIDMMMMMMM
TNK2          MMMMIDIDMMMMMMMM
HINT1         MMMMDMMMMMMMMMI
EIF4H         MMMMDDMMMMMMMMVV
EIF4A1        MMMMIDMMMMMMMMM
FAM171A1      MMMMIDIDMMMMMMMM
ZNF638        MMMMIIDDMMMMMMMM
STARD4-AS1    MMMMIDIDMMMMMMMM
PDHX          MMMMIDIDIDMMMMMMM
MICAL3        MMMMIIDDMMMMMMMM
NR4A3         MMMMIDMMMMMMMMM
AC007376.2    MMMMDIDIDDMMMMMMII
RPL15         MMMMIDMMMMMMMMM
PLEKHA2       MMMMIIDDMMMMMMMM
PRKD1         MMMMIDMMMMMMMMM
UQCRFS1       MMMMIIDDIDIDMMMMMM
NR3C1         MMMMDDMMMMMMMMVV
LDHB          MMMMIDIDIDIDMMMMMM
AP2M1         MMMMIDIDIDMMMMMMM
COX6A1        MMMMDDMMMMMMMMVV
PARK7         MMMMIIDDIDMMMMMMM
SSBP3         MMMMIDIDMMMMMMMM
CYC1          MMMMDDDMMMMMMMVVV
PHB           MMMMDDMMMMMMMMVV
SLMAP         MMMMDDMMMMMMMMVV
MAPRE1        MMMMDIDMMMMMMIMM
HBA1          MMMMIDIDMMMMMMMM
SORBS1        MMMMDMMMMMMMIMM
PSMB6         MMMMIDIDIDMMMMMMM
ATP5F1A       MMMMIIIDDDIDMMMMMM
MFSD4B        MMMMIDIDMMMMMMMM
PHF14         MMMMIDMMMMMMMMM
SMAP1         MMMMIIDDIDMMMMMMM
ECI1          MMMMIIDDIDMMMMMMM
PFN1          MMMMDDMMMMMMMMVV
NFE2L1        MMMMIIDDIDMMMMMMM
EIF1AX        MMMMIIDDMMMMMMMM
CYCS          MMMMDDMMMMMMMMII
AFAP1L1       MMMMIDIDMMMMMMMM
FOXK1         MMMMIDIDMMMMMMMM
SSPN          MMMMIIDDMMMMMMMM
RBMS1         MMMMDDMMMMMMMMVV
A4GALT        MMMMIIDDMMMMMMMM
IQSEC1        MMMMIDMMMMMMMMM
CLEC2D        MMMMDMMMMMMMIMM
VPS53         MMMMIDIDMMMMMMMM
MT-ND4L       MMMMDIDMMMMMMIMM
MT-ND3        MMMMDIDIDMMMMMMMI
CALM2         MMMMDDMMMMMMMMVV
RPL17         MMMMDDMMMMMMMMII
PPIA          MMMMDDMMMMMMMMII
PXDNL         MMMMDDMMMMMMMMII
PHF3          MMMMIDMMMMMMMMM
TBX5          MMMMDDMMMMMMMMVV
DACH1         MMMMIDMMMMMMMMM
TRAK1         MMMMIDIDIDMMMMMMM
LEMD3         MMMMDDMMMMMMMMVV
C7orf50       MMMMIDIDIDMMMMMMM
DNM2          MMMMIIDDMMMMMMMM
PSMD8         MMMMIDMMMMMMMMM
ACOX3         MMMMIDIDMMMMMMMM
DECR1         MMMMIIDDIDMMMMMMM
SET           MMMMIDMMMMMMMMM
GEM           MMMMDMMMMMMMIMM
MAP4K4        MMMMIDIDIDMMMMMMM
CSRP2         MMMMDDMMMMMMMMVV
AL139317.5    MMMMIIDDMMMMMMMM
CCNI          MMMMIDIDIDMMMMMMM
KCTD1         MMMMIDIDIDIDMMMMMM
NME4          MMMMIDIDMMMMMMMM
DDHD1         MMMMDDMMMMMMMMVV
CPSF6         MMMMIDIDMMMMMMMM
MTR           MMMMIIDDMMMMMMMM
TNRC18        MMMMIDMMMMMMMMM
ADCY5         MMMMIDIDMMMMMMMM
MRPL34        MMMMIDMMMMMMMMM
RAMP1         MMMMDDMMMMMMMMII
TMEM161B-AS1  MMMMIIDDMMMMMMMM
DARS          MMMDDMMMMMMMMMVV
DYNC2H1       MMMIIIDDDMMMMMMMM
CLCN6         MMMIDIDMMMMMMMMM
ANXA3         MMMDDIDIDMMMMMMMVV
PGRMC1        MMMIDIDIDIDMMMMMMM
VCP           MMMDDIDMMMMMMMMVV
EPN2          MMMDDMMMMMMMMMII
LRPPRC        MMMIDIDIDMMMMMMMM
ERH           MMMDDDMMMMMMMMVVV
ANKRD18A      MMMIIDDIDMMMMMMMM
BMPR1A        MMMIDIDIDMMMMMMMM
RTRAF         MMMDDMMMMMMMMMVV
ATG2B         MMMIIIDDDMMMMMMMM
ETNK1         MMMIIDDIDMMMMMMMM
AHNAK         MMMDDMMMMMMMMMVV
PRDX2         MMMDDDMMMMMMMMVVV
GPX4          MMMDDDMMMMMMMMVVV
RANBP17       MMMIDIDIDIDMMMMMMM
ACTN1         MMMIDIDMMMMMMMMM
COLQ          MMMIDIDIDMMMMMMMM
RBM8A         MMMDDDMMMMMMMMVVV
EPS15L1       MMMDDMMMMMMMMMVV
XIST          MMMIDIDIDIDMMMMMMM
LMO7          MMMIDIDMMMMMMMMM
CAP2          MMMIDIDIDMMMMMMMM
VCL           MMMIDIDIDMMMMMMMM
CADPS         MMMDDDMIIIMMMMMMM
SKP1          MMMDDDMMMMMMMMVVV
PKM           MMMIDIDIDIDMMMMMMM
HK2           MMMIDDIDMIMMMMMMM
GADD45G       MMMIIDDIDIDMMMMMMM
SCAF11        MMMIIDDMMMMMMMMM
RASD1         MMMIIDDIDMMMMMMMM
ELOVL5        MMMIIDDIDIDMMMMMMM
SLC25A3       MMMDDDMMMMMMMMVVV
GOLGA4        MMMIIDDIDIDMMMMMMM
PRKAR1A       MMMDMMMMMMMMIMM
HSPA5         MMMIDIDIDMMMMMMMM
FRAS1         MMMDDDMMMMMMMMVVV
PPP1CB        MMMIDMMMMMMMMMM
SNRPG         MMMIDIDIDIDMMMMMMM
NCL           MMMIDIDMMMMMMMMM
DANCR         MMMIDIDIDMMMMMMMM
C9orf72       MMMDDDIDMMMMMMMVVV
UBE2O         MMMDIDMMMMMMMIMM
PTPRD         MMMIDIDIDMMMMMMMM
AC087286.2    MMMIIDDIDIDMMMMMMM
H3F3A         MMMDDDMMMMMMMMIII
H3F3B         MMMDDDMMMMMMMMVVV
FAM110B       MMMDIDIDMMMMMMIMM
ARL17B        MMMIDDMMMMMMMIMM
GNAQ          MMMIDIDIDMMMMMMMM
BTG1          MMMIDIDIDIDMMMMMMM
DYRK2         MMMDDMMMMMMMMMVV
DNAJA1        MMMIIDDIDIDMMMMMMM
RBM24         MMMDDMMIIMMMMMMM
LINC01278     MMMDDMMMMMMMMMVV
OGA           MMMDDDMMMMMMMMVVV
TCEAL9        MMMIDIDIDMMMMMMMM
RPLP2         MMMIDIDMMMMMMMMM
PTPN21        MMMDDDMMMMMMMMVVV
UQCRC1        MMMDDDMMMMMMMMVVV
CNOT11        MMMDDDMMMMMMMMVVV
PPIB          MMMDDDMMMMMMMMVVV
NAV2-AS2      MMMIDIDIDIDMMMMMMM
WLS           MMMDDMMIIMMMMMMM
PSD3          MMMDMMMMMMMMIMM
JPH2          MMMDDDMIIIMMMMMMM
PTMS          MMMDDDMMMMMMMMVVV
NOP10         MMMIIDDIDIDMMMMMMM
ASAH1         MMMIDIDIDMMMMMMMM
DCUN1D2       MMMDMMMMMMMMIMM
NIN           MMMIDIDIDIDMMMMMMM
TMEM135       MMMIIDDIDMMMMMMMM
PSMD1         MMMDDDMMMMMMMMVVV
COL2A1        MMMDDDMMMMMMMMVVV
FBH1          MMMIIDDIDMMMMMMMM
CFL1          MMMDDDMMMMMMMMVVV
PSMB1         MMMDDDMMMMMMMMVVV
RPL18         MMMIIDDIDIDMMMMMMM
CNBP          MMMDDMMMMMMMMMVV
RAB20         MMMIDIDIDIDMMMMMMM
SPCS1         MMMIDIDIDIDMMMMMMM
PLEKHJ1       MMMDIDDDMMMMMMMVVV
LINC01505     MMMDDDMMMMMMMMVVV
TAGLN         MMMIIDDIDMMMMMMMM
HSPE1         MMMIDDIDMMMMMMMMI
LMBRD2        MMMDDDMMMMMMMMVVV
SMARCE1       MMMIIDDIDIDMMMMMMM
IRF2BP2       MMMIDIDMMMMMMMMM
CERS6         MMMIIDDMMMMMMMMM
DDT           MMMDDDMMMMMMMMVVV
PHACTR3       MMMIIMMMMMMMMDDM
SMTNL2        MMMDDDMMMMMMMMVVV
SLC25A4       MMMDIDDMMMMMMMMII
PSMD14        MMMDDMMMMMMMMMVV
LMAN2         MMMIDIDIDIDMMMMMMM
FAM162A       MMMIIDDMMMMMMMMM
FHIT          MMMDDMMMMMMMMMVV
DENND4A       MMMIDIDIDIDMMMMMMM
IGF1R         MMMDDDMMIIIMMMMMM
PSMB5         MMMDDDMMMMMMMMVVV
SUCO          MMMIDDMMMMMMMIMM
HNRNPK        MMMDDMMMMMMMMMVV
CKAP4         MMMDDDMMMMMMMMVVV
MDH1          MMMDDDMMMMMMMMIII
SLC8A1-AS1    MMMIDIDIDMMMMMMMM
NR2F2         MMMDDIDMMMMMMMMVV
MPP7          MMMIDIDIDMMMMMMMM
SLC25A27      MMMIIDDIDMMMMMMMM
MPDZ          MMMDDMMMMMMMMMVV
SPIRE1        MMMIDIDIDMMMMMMMM
ANKRD1        MMMIDDMMMMMMMMMI
ATP5MC3       MMMDDDMMMMMMMMVVV
SPPL3         MMMIIDDIDMMMMMMMM
ST6GALNAC3    MMIDDDDMMMMMMMMVVV
NES           MMDDDMMMMMMMMMVVV
VPS37B        MMIIMMDIDIDDMMMMMM
AKAP6         MMIIIMMDDDMMMMMMM
PPP1R1A       MMIIIDDIDIDIDDMMMMMM
AFF2          MMDDDIDMMIDIIIMMMMM
KCNIP4        MMIDDDIDIDMMMMMMMVV
STK38         MMIDIDIDIDMMMMMMMM
GPRC5B        MMDDDDIDMMMIIIIMMMM
VDAC1         MMDDDMMIIIMMMMMMM
CLDN6         MMIIMMMMMMDDMMMM
MDGA2         MMIIIDIDIDIDDDMMMMMM
NPM1          MMIMMIDDIDMMMMMMM
NPAS2         MMIDIDIDIDMMMMMMMM
IQGAP1        MMIDDMMMMMMMMIMM
PLXNC1        MMIIIIIDIDIDIIDDDDDMMMD
ACTN4         MMIDDDIDIDMMMMMMMVV
BRINP3        MMIIMIDIDIDIIIDDDDMMMD
USP6NL        MMIDIDIDIDMMMMMMMM
CAMKMT        MMDDMMMMMMMMMMVV
NPPB          MMIMMIDDMMMMMMMM
MT-ND4        MMIIDDIDIDIDMMMMMMM
PCDH9         MMDDDMMMMMIIIMMMM
FGF12         MMIDDDDMMMMMMMMVVV
NEDD9         MMDDDIDIDDMMMMMVVVVVD
PSAP          MMDDDMMMMMMMMMVVV
ARHGAP10      MMDDDDMMMMIIIIMMMM
GAN           MMIIDDIDIDIDMMMMMMM
DTNA          MMIIDDIDIDIDMMMMMMM
SYNDIG1       MMIDDDDIDMMMMMMVVVVD
JARID2        MMIIDDIDMMMMMMMMM
HNRNPA1       MMIDDDDMMMMMMMMVVV
NCKAP5        MMDDMMMMMMMMMMVV
MMP23B        MMIIDMMMMDMMMMMM
TSTD3         MMIDIDIDIIDDMMMMMMM
MYH7          MMDIDIDMMMMMIMMMM
ABTB2         MMIDIDIDIDIDMMMMMMM
NUAK1         MMIIIIDIDIDIDDDDMMMMM
EIF3F         MMIMMDMMMMMMMMM
ARHGAP35      MMDDDMMMMMMMMMVVV
CREB3L2       MMDDMMMMMMMMMMVV
VDAC2         MMDDDMMMMMMMMMVVV
RTN4          MMIDDDIDMMMMMMMMVV
GNG7          MMIDDDIDMMMMMMMMVV
RAC1          MMDDDMMMMMMMMMVVV
AKAP13        MMIDDMMIDIDIIDMMMMM
FDFT1         MMIIDIDIDIDDMMMMMMM
STAT4         MMIIDDIDMMMMMMMMM
MLLT3         MMIMMDIDMMMMMMMM
PDE8B         MMIMMDIDIDMMMMMMM
SMG1          MMIIMMDDMMMMMMMM
TUSC3         MMIMMDMMMMMMMMM
FGF13         MMIIIMDDDMMMMMMMM
PWRN1         MMDMDDMMMMMMMMVVV
RIC8B         MMIIDDIDMMMMMMMMM
SHOX2         MMIDDMMMMMMMMMMI
ALDOC         MMIIDDIDIDIDMMMMMMM
TENM2         MMDDMMMMMMMMMMVV
KLHL24        MMIDIDIDIDMMMMMMMM
CRTC3         MMIIMMDDMMMMMMMM
MSMO1         MMIIIIMDDDDMMMMMMM
ARFGEF2       MMDDIDMMMMMMMMMVV
AC092640.1    MMIDIDIDMMMMMMMMM
RPL41         MMIDDDMMMMMMMMMII
PLCB4         MMIDIDIDIDIDMMMMMMID
BCAR3         MMIIDDIDIDIDMMMMMMM
RPL36A        MMIDIDIDMMMMMMMMM
PGK1          MMIDDDIDMMMMMMMMVV
RNGTT         MMDDIDMMMMMMMMMVV
SQSTM1        MIMMMMDMMMMMMMM
RPL10A        MIIMMMDDMMMMMMMM
CNTN4         MIIDDIDIDMMMMMMMMM
PIP5K1B       MDDDMMMMMMMMMMVVV
PDK1          MDMMMMMMMMMMIMM
MCRIP1        MIIMMMMDDMMMMMMM
LSAMP         MDMMDMMMMMMMMMVV
NFASC         MDIVVVVVDDDDDMMMMMMM
KSR1          MIIIIDIDIDIDDDDMMMMMM
A2M           MIIIDIDIDIIDDDDMMMMMM
EIF2AK3       MIMMDDIDMMMMMMIMM
ITSN2         MDDDDDMIIIIDIIMMMMMM
ADAM23        MDDDDDMMIIIIIMMMMMM
RPL26         MIIIMMMDDDMMMMMMM
NR2F1         MIIIIDDDIDIDDMMMMMMM
ITM2C         MIIIMMDDDMMMMMMMM
UQCRH         MIIMMMMDDMMMMMMM
ANO5          MIIIDIDIDIIDDDDMMMMMM
PRKG1         MIDDDDMMIIIMMMMMMM
PIM1          MIMMMMDMMMMMMMM
RPL28         MIMMMDIDMMMMMMMM
TUBE1         MIDIDIDIDIDMMMMMMMM
TBCA          MIMMMMDIDMMMMMMM
CCND2         MIMMMMDMMMMMMMM
AFG3L2        MDDDMMMMMMMMMMVVV
NDUFB5        MIMMMMDIDMMMMMMM
NOP53         MIMMMDMMMMMMMMM
PLEKHG4B      MDDMDIDMMMMMMMMVVV
RPS29         MIIMMMDDMMMMMMMM
CCT2          MIIMMMMDDMMMMMMM
MYO18B        MIMMMDIDMMMMMMMM
TPP2          MIIIIMMDDDDMMMMMMM
DOCK9         MIIIMMMDDDMMMMMMM
AC006460.2    MDMMMIDIMMMMMMMM
MAP4          MIIMMMDDMMMMMMMM
ARL6IP5       MIIMMMDIDDMMMMMMM
HMBOX1        MDDDDDMMIIIIIMMMMMM
RPL14         MIIMMDDMMMMMMMMM
ATF3          MIIDDIDMMMMMMMMMM
SLC47A1       MDMMMMMVVVDDMMMMM
TET2          MDDDMMMMMMMMMMVVV
NDUFC1        MIIMMMDIDDMMMMMMM
NRG3          MIIIIDIDIDDDDMMMMMMM
RPS19         MIIMMMDIDDMMMMMMM
METTL26       MIMMDIDIDIDMMMMMMM
PCBP1         MIIMMMDDMMMMMMMM
RELN          MDDDMMIIIMMMMMMMM
H2AFY         MIIIIDIDIIDDDDDMMMMMM
MAF           MIIIIIIDIDIDDDDDDMMMMM
GRIA1         MIIIIIMMMMMMMMDDDDD
CARMIL1       MIDDIDMMMMMMMMIMM
KCNMB2-AS1    MIIIIDIDIDIIIIIIIDDDDDDDDDD
GABRB2        MIMMDIDIDMMMMMMMM
HEY1          MIIIIMMMMMMMMMWWWW
AC092353.2    MIIDDDIDIDMMMMMMIMM
CCNL1         MDMMIDIDIMMMMMMMM
UQCC2         MDDMMMMMMMMMMMVV
NEDD8         MDMMMMMIMMMMMMM
EEF1D         MIIMMMMDDMMMMMMM
CCND1         MIDDDMMMMMMMMMMVV
PNPLA7        MIDDDIDMMMMMMMMMII
DKK3          MIIIDIDIDIDDDMMMMMMM
RPL31         MIIMMMMDDMMMMMMM
FGFR1         MDMMMIMMMMMMMMM
TXNIP         MDMMMMMMMMMMIMM
TRPM3         MIIIIIDIDIDIDDDDDMMMMM
UBA52         MIIMMMDIDDMMMMMMM
RPE           MIIIIDIDIDIDDDDMMMMMM
AFF1          MDMMMMMIMMMMMMM
PNKD          MIIMMMMDDMMMMMMM
SIK2          MDDMMIDIDIIMMMMMMM
AL356737.2    MIIIIIMMDDDDDMMMMMM
GNAL          MIIIDIDIDIDIIDDDDMMMMM
MCM6          MIIMMMMMMDDMMMMM
BVES          MDMDIDDMMIIIMMMMMM
CDH12         MIIDDIDIDIDDDMMMMMMVV
RPL27         MIIIMMMDDDMMMMMMM
EEF1G         MIIMMMDIDIDDMMMMMM
SNU13         MIIIMMDDDMMMMMMMM
WDR62         DMMMMMMMMMMVVVVDDD
Z94721.1      DMMMMMMVVVVVVVIDWWWWWW
AL022097.1    DMMMMMMVVVVVVVIDWWWWWW
SLC5A12       DMMMMMMMMMMMMIID
TRPC4         DMMMMMMMMMMMMIDI
BMP2          IMMMMMMMMWWWIIMMM
PTGDS         DMMMMMMMMMIMMMM
SEMA5A        IMMVVVVVVDDDDDDDMMMMM
DLGAP1        IMWWWWWWWIIIIIIIIIIIIDDDDDD
ST3GAL6       IMMVVVVVDDDDDDMMMMMM
RFX2          DMMMMVVVDDMMMMMMM
MYCN          IMMMMMMMIDWWWWIIIMM
SPRY4-AS1     IMMVVVVDDDDDMMMMMMM
WASHC2A       IMMVVVVDIDDDDDMMMMMM
MAP3K14       IMMVVVVDDDDDMMMMMMM
MTRNR2L12     IMMMMMDDMMMMMIMM
HECTD4        IMMMMMDIDMMMMMMM
CFLAR         IMMMMMDMMMMMMMM
USP22         IMMMMMDIDMMMMMMM
CTNND1        IMMMMMDMMMMMMMM
CEP95         IMMMMMDMMMMMMMM
PIP4K2A       IMMMMMDMMMMMMMM
GRB10         IMMMMMDMMMMMMMM
MRPS24        IMMMMMIDDMMMMMMM
PPP6R3        IMMMMMDMMMMMMMM
ATP5PB        IMMMMMDIDMMMMMMM
UTY           DMMMMIIIVVVVVVVDDDDDDDDD
PSMC5         IMMMMDIDMMMMMMMM
PSMA7         IMMMMDMMMMMMMMM
SNX24         IMMMMDMMMMMMMMM
C1orf122      IMMMMDMMMMMMMMM
MLEC          IMMMMDMMMMMMMMM
SH3PXD2A      IMMMMDMMMMMMMMM
EIF5A         IMMMMDMMMMMMMMM
TMED9         IMMMMDMMMMMMMMM
PGM1          IMMMMDIDMMMMMMMM
SH3BGR        IMMMMDIDMMMMMMMM
PPM1K         DMMMMIDIDIIDMMMMMM
SYNPO         IMMMMDIDMMMMMMMM
EML4          IMMMMDIDMMMMMMMM
MAZ           IMMMMDMMMMMMMMM
PRKAR2B       IMMMMDIDIDMMMMMMM
CPNE5         IMMMMDIDMMMMMMMM
MYO1E         IMMMMDIDMMMMMMMM
PRMT1         DMMMMIIDIDMMMMMMM
HBG2          IMMMMDMMMMMMMMM
RILPL1        IMMMMDIDMMMMMMMM
ADCY6         IMMMMDIDMMMMMMMM
ELOC          IMMMMDMMMMMMMMM
PDZD2         IMMMMDIDMMMMMMMM
PDLIM3        IMMMMDIDMMMMMMMM
HSP90AB1      IMMMMDIDMMMMMMMM
MEIS2         IMMMMDIDIDMMMMMMM
PMVK          IMMMMDMMMMMMMMM
DIP2C         IMMMMDIDMMMMMMMM
TRPM7         IMMMMDIDMMMMMMMM
SCN5A         IMMMMDIDMMMMMMMM
BZW2          DMMMMIIDIDMMMMMMM
TRIR          IMMMMIDIDDMMMMMMM
MT-ATP8       IMMMMDDMMMMMMIMM
GJA1          DMMMMIIDMMMMMMMM
MARK3         IMMMMDIDMMMMMMMM
LINC00881     IMMMMDDMMMMMMIMM
RPL21         IMMMMDIDIDMMMMMMM
OXA1L         IMMMMDMMMMMMMMM
UBB           DMMMMIIDIDMMMMMMM
MPC1          IMMMMDIDMMMMMMMM
GOT1          IMMMIDIDIDDMMMMMMM
AGPAT5        DMMMIDIDIMMMMMMMM
UBC           IMMMDDDMMMMMMMMII
KIF9-AS1      IMMMDIDIDMMMMMMMM
PDE3A         DMMMIDIDMMMMMMIMM
TIAM1         IMMMDIDMMMMMMMMM
MAST4         DMMMIIDIDMMMMMMMM
PLXNA2        IMMMIDIDIDIDDMMMMMM
TCEAL4        IMMMDDDDMMMMMMMVVV
MPP3          IMMMDDDMMMMMMMMII
FBXL7         IMMMDIDIDMMMMMMMM
MLLT11        IMMMIDIDIDDMMMMMMM
HBA2          IMMMDIDMMMMMMMMM
FSTL4         IMMMIDIDIDDMMMMMMM
MEF2C         DMMMIDIMMMMMMMMM
FAM78A        IMMMIDIDIDIDDMMMMMM
CDCA7L        IMMMDIDIDMMMMMMMM
PGAM1         IMMMDDDMMMMMMMMVV
CRYBG3        IMMMDDDMMMMMMMMVV
AUH           IMMMDIDIDMMMMMMMM
SLC2A13       IMMMDDDMMMMMMMMII
DPF3          IMMMDDDMMMMMMMIIID
AC093849.2    IMMMDIDIDIDMMMMMMM
PFKP          IMMMDIDIDMMMMMMMM
STMN1         IMMMDDDMMMMMMMMVV
MARCH3        DMMMDDMMMMMMMMIII
HSP90AA1      IMMMDIDIDIDMMMMMMM
ARHGEF12      DMMMIIDIDMMMMMMMM
PPIC          IMMMDDDMMMMMMMMVV
MAGED2        IMMDDDDMMMMMMMMVVV
GSTP1         IMMDDDDMMMMMMMMVVV
LRRC52-AS1    IMMIDIDIDIDDMMMMMMM
MDK           IMMDIDDDMMMMMMMMVV
SPINT2        IMMIDIDIDIDDMMMMMMM
GPATCH8       IMMDDDMMMMMMMMMVV
ADGRD1        IMMDDDDMMMMMMMMVVV
AL049828.1    DMMIDIDIDIMMMMMMMM
DPYSL3        IMMDIDIDDMMMMMMIMM
CNN3          IMMDDDMMMMMMMMMVV
LIFR          IMMDIDDMMMMMMMIMM
LINC02268     IMMDIDDDMMMMMMMMVV
IL1RAPL1      DMMDDMMIIIMMMMMMM
ATP2B4        DMMDIDIDMMMMMMMMVV
ZBTB16        IMMDDDDMMMMMMMMVVV
FERMT2        IMMDIDIDMMMMMMMMM
TNNI3K        IMMDDDDMMMMMMMMIII
HSPB7         DMMIMMIIDDMMMMMMM
CBX5          IMMDDDDMMMMMMMMVVV
ACO2          IMMDDDMMIIMMMMMMM
ZBTB20        IMDDMMMMMMMMMIMM
PLXDC2        IMIIMMIDDDDMMMMMMM
NEK10         IMIIMMDDDMMMMMMMM
CSRNP3        DDMMMMMMMMMMMMVV
PSMA2         DDMMMMMMMMMMMMVV
AC098617.1    DDMMMMMMMMMMMMVV
RLF           DDMMMMMMMMMMMMVV
HADHB         DDMMMMMMMMMMMMVV
CDCA8         DDMMMMMMMMMVVVVVDDD
NUP35         IIMMMMMMMMMMMMWW
PARM1         DDMMMMMMMMMMMMVV
ZCCHC8        DDMMMMMMMMMMMMVV
MFN2          DDMMMMMMMMMMMMVV
TTN-AS1       DDMMMMMMMMMMMMVV
TBX18         DDMMMMMMMMMMVVVVDD
GNG11         IDMMMMMMMMMMMMM
MLLT6         IDMMMMMMMMMMMMM
AL603840.1    IDMMMMMMMMMMMMM
RPL22L1       IDMMMMMMMMMMMMM
PROX1         IDMMMMMMMMMMMMM
VEGFA         IDMMMMMMMMMMMMM
NORAD         IDMMMMMMMMMMMMM
PRDX6         IDMMMMMMMMMMMMM
PSMC3         IDMMMMMMMMMMMMM
SGCG          IDMMMMMMMMMMMMM
SPECC1L       IDMMMMMMMMMMMMM
KDM2A         IDMMMMMMMMMMMMM
CAMTA1        IDMMMMMMMMMMMMM
KIAA1217      IDMMMMMMMMMMMMM
DYRK3         IDMMMMMMMMMMMMM
PNRC1         IDMMMMMMMMMMMMM
UQCRC2        IDMMMMMMMMMMMMM
FIGN          IDMMMMMMMMMMMMM
MAPRE2        IDMMMMMMMMMMMMM
MSI2          IDMMMMMMMMMMMMM
LGALS8        IDMMMMMMMMMMMMM
PPFIA4        IDMMMMMMMMMMMMM
C1orf198      IDMMMMMMMMMMMMM
DNAH14        IDMMMMMMMMMMMMM
OIP5-AS1      IDMMMMMMMMMMMMM
ESYT2         IDMMMMMMMMMMMMM
SLC6A6        DDMMMMMMMMMMMMII
LMOD1         DDMMMMMVVVVDDMMMMM
FEN1          DDMMMMMVVVVIDWWWWWWIIII
RTKN2         DDMMMMMVVVVDDMMMMM
EPAS1         DIVVVVVVDDDDDDMMMMMMM
SORCS2        IDMMVVVVDIDDDDMMMMMM
ALKBH7        DDMMMMMIIMMMMMMM
MT-CO1        IIMMMMMDDMMMMMMM
RPL3          IIMMMMMDDMMMMMMM
BTBD9         DDMMMMMIIMMMMMMM
MYO9B         DDMMMMMIIMMMMMMM
AC138305.1    IIMMMMMDDMMMMMMM
TPM2          IIMMMMMDDMMMMMMM
LNPEP         IIMMMMDDMMMMMMMM
ADAMTS6       IIMMMMDIDDMMMMMMM
PPARGC1A      IIMMMMDDIDMMMMMMM
ARL15         IIMMMMDDMMMMMMMM
PPP1CC        IIMMMMDDMMMMMMMM
CRPPA         IIMMMMDDMMMMMMMM
RHEB          IIMMMMDDMMMMMMMM
GABARAP       IDMMMMIDIDMMMMMMM
SRSF2         IIMMMMDDMMMMMMMM
RAP1GAP2      IIMMMMDDMMMMMMMM
FRMD3         IIMMMMDDMMMMMMMM
MT-ND5        IIMMMMDDMMMMMMMM
NR4A1         IIMMMMDDMMMMMMMM
BABAM2        IIMMMMDDMMMMMMMM
MPRIP         IIMMMMDDMMMMMMMM
SOX5          IDMMMMDMMMMMMIMM
TBL1XR1       IIMMMMDDMMMMMMMM
UBE3A         IIMMMMDDMMMMMMMM
USP9Y         DDMMMMDMMMMMMIIIID
FAM53B        IIMMMMDDMMMMMMMM
CCN1          IIMMMMDDMMMMMMMM
SLC25A5       IDMMMMIDIDMMMMMMM
ST6GAL1       IIMMMMDDMMMMMMMM
SAP30         IIMMMMDDMMMMMMMM
OGDH          IIMMMMDDMMMMMMMM
DHX15         IIMMMMDDMMMMMMMM
NDUFS1        IIMMMMDDMMMMMMMM
HSPB8         IIMMMMDDMMMMMMMM
NFYC          IIMMMMDDMMMMMMMM
PIK3R1        IDMMMMDDMMMMMMMVV
SERBP1        IIMMMMDDMMMMMMMM
STRN3         IIMMMMDDMMMMMMMM
RPL6          IIMMMMDDMMMMMMMM
PDIA3         IIMMMMDDMMMMMMMM
PAFAH1B1      IIMMMMDDMMMMMMMM
RPL19         IIMMMMDIDDMMMMMMM
VPS35L        IIMMMMDDMMMMMMMM
WNK2          IIMMMMDDMMMMMMMM
HIPK3         IIMMMMDDMMMMMMMM
MIA2          IIMMMMDDMMMMMMMM
HIVEP2        IIMMMMDDMMMMMMMM
COX7A2        IDMMMMIIDDMMMMMMM
JAG1          IIMMMMDDMMMMMMMM
HIF3A         IIMMMMDIDDMMMMMMM
HHATL         IIMMMMDDMMMMMMMM
PDHA1         IDMMMMIDIDMMMMMMM
SUMO2         IDMMMMIIDIDDMMMMMM
FNBP4         IIMMMMDDMMMMMMMM
IDH2          IDMMMMDDMMMMMMMVV
MT-ND1        IIMMMMDDMMMMMMMM
HSPB1         IIMMMMDDMMMMMMMM
HSPA8         IIMMMMDDIDMMMMMMM
SRP14         IDMMMMIDMMMMMMMM
XPO1          IIMMMMDDMMMMMMMM
RTL8C         IDMMMMIIDDMMMMMMM
CHN1          IIMMMMDDMMMMMMMM
TRRAP         IIMMMMDDMMMMMMMM
HMGN3         IDMMMIDMMMMMMMMM
SYNPO2L       IDMMMIIDIDDMMMMMMM
TMBIM6        DDMMMIDIDIIMMMMMMM
BEX3          IDMMMDDMMMMMMMMVV
PAGR1         IDMMMDDMMMMMMMMVV
STARD3NL      IIMMMDIDIDDMMMMMMM
MYL9          IDMMMDDMMMMMMMMVV
YWHAB         IDMMMIIDIDDMMMMMMM
PLOD2         IDMMMDIDMMMMMMIMM
TECRL         IIMMMIDIDIDDMMMMMMD
HDAC2         IDMMMDDMMMMMMMMVV
APOBEC2       DDMMMIIIDIDMMMMMMM
NDUFA13       IDMMMDDMMMMMMMMII
PCMTD1        IIMMMDDMMMMMMMMM
LINC01954     IDMMMDMMMMMMMIMM
TBX20         IIMMMDDDMMMMMMIMM
SNHG6         IIMMMDDIDIDMMMMMMM
DSTN          IIMMMDDMMMMMMMMM
COL4A2        IIMMMDDDMMMMMMIMM
OST4          IIMMMDDMMMMMMMMM
GFOD1         IIMMMDIDDMMMMMMMM
ENO3          IDMMMIIDIDIDDMMMMMM
SSBP2         DDMMMDMMMMMMMMVVV
TCEAL8        IDMMMDDMMMMMMMMVV
TPI1          IDMMMDIDDMMMMMMMII
SLC25A6       IDMMMDDMMMMMMMMVV
TUBB          IDMMMDDMMMMMMMMVV
MYL6          IDMMMDDMMMMMMMMII
KLF12         IDMMMIDMMMMMMMMM
PPM1B         IIMMMDDMMMMMMMMM
VWDE          IIMMMDIDDMMMMMMMM
KDM4C         DDMMMIDIIMMMMMMMM
COL18A1       IDMMMDDMMMMMMMMVV
EID1          IDMMMDMMMMMMMMMI
ANKRD11       IIMMMDIDDMMMMMMMM
DIP2B         IDMMMIDIDMMMMMMMM
CORIN         IIMMMIDIDDDMMMMMMM
TSC22D2       IDMMMIDIDMMMMMMMM
HDAC9         IIMMMDIDDMMMMMMMM
MTHFR         IIMMMIDIDIDDDMMMMMM
FBXL17        DDMMMIDIDIIMMMMMMM
RNF149        IIMMMDIDDMMMMMMMM
ADAMTS9       IIMMMDIDIDDMMMMMMM
CFL2          IDMMMDDMMMMMMMMVV
TRPS1         IDMMMDDMMMMMMMMVV
DSG2          IIMMMDDDDMMMIIMMMM
RPSA          IDMMMIDMMMMMMMMM
PA2G4         IDMMMDDMMMMMMMMVV
MCRS1         IIMMMDIDIDDMMMMMMM
KRT8          IDMMMDDMMMMMMMMVV
RORA          IIMMMIDIDIDDDMMMMMM
PTOV1         IDMMMIDIDIDMMMMMMM
CACNA1D       IIMMMDIDIDDMMMMMMM
PHC2          IIMMMIDIDDDMMMMMMM
FKBP3         IDMMMIDIDIDMMMMMMM
AC040174.1    IDMMMIDIDIDMMMMMMM
PACS1         IDMMMIDIDMMMMMMMM
EPB41L4A      IDMMMDDMMMMMMMMVV
MGST3         IDMMMIIDIDDMMMMMMM
EEF1A2        IDMMMIDIDMMMMMMMM
ADAMTS1       IIMMMDDDDMMMMMMMVV
DAPK3         IDMMMIDMMMMMMMMM
ALPK3         IDMMMDIDMMMMMMIMM
ARRDC3-AS1    IIMMMDIDIDDMMMMMMM
PPP1R14C      IIMMMDDMMMMMMMMM
COX7A2L       IDMMMIDIDIIDDMMMMMM
ATP5ME        IDMMMIDMMMMMMMMM
HCN1          DDMMMIIIDIDMMMMMMM
PCGF5         IIMMMDDMMMMMMMMM
CKMT2         IIMMMDIDIDDMMMMMMM
STAT3         DDMMMIDIIMMMMMMMM
LINC02248     DDMMIDIDIIIIDDDMMMMIID
RYR3          DDMMIDIDIIIIDDMMMMMM
CD164         IIMMDIDDMMMMMMMMM
USP28         DDMMIDIMMMMMMMIMM
TUFM          DDMMDMMMMMMMMMVVV
TNNI3         IIMMIIDIDDDDMMMMMMM
DST           DDMMDDMMIIIIMMMMMM
AC004540.1    IIMMIDIDIIIDDDDDMMMMM
BRWD1         IIMMDDIDIDMMMMMMMM
PPP2R3A       IDMMDDMMMMMMMMMVV
KCNH7         DDMMIDIDIIMMMMMMMM
RASGRP3       DDMMIIMMMMMMMMMM
KCNQ5         IIMMIMMIIDDDDMMMMMD
MYH9          IDMMIDIDIDDDMMMMMMII
LRIG1         IIMMDIDIDDMMMMMMMM
COL21A1       IIMMIDIDIDDDMMMMMMM
CAMK2B        IDMMIDIDIDMMMMMMMM
ASPH          DDMMIDIIMMMMMMMMM
PLCL1         IIMMIDIDIIIIIIIIDDDDDDDDDD
ZMAT4         IDMMDDIDMMMMMMMMVV
TBX3          IIMMIDIDIDDDMMMMMMM
ARID5B        IIMMDDDDMMMMMMMMVV
TSHZ2         IDMMIDIDIIIDDDMMMMMID
KIAA0232      IIMMIDIDIDDDMMMMMMM
KALRN         IIMMDIDIDDDMMMMMIMM
RPS28         IDMMDDMMMMMMMMMII
LARGE1        DDMMIDIDMMMMMMIIMM
SQLE          IDMMIDIDIIDDMMMMMMM
DNAJB6        IDMMIDIDMMMMMMMMM
PANK3         IIMMDIDIDDMMMMMMMM
UGP2          IDMMIMMIDDMMMMMMM
NPPA          DDMMIIIIDDMMMMMMMM
ADAMTS19      DDMMDMMMMMMMMMVVV
AP005329.2    IIMMDDMMMMMMMMMM
SUCLG1        IIMMIDIDIDDDMMMMMMM
ATP11A        DDMMIIDIDIMMMMMMMM
SRSF10        DDMMIDIIMMMMMMMMM
DRAP1         DDMMDMMMMMMMMMVVV
ITGA9         IIMMDDDDDMMMIIIMMMM
CPEB4         IIMMDIDDMMMMMMMMM
PURG          IDMMIDIMMMMMMMDMM
MBNL2         IDMMDDMMMMMMMMMVV
GALNT17       DDMMIDIDMMMMMMMMII
MAP3K20       DDMDDMMMIIIIMMMMMM
CNTN5         IDMIIDIDIIIIDDDDDMMMMM
SLIT3         IIMIIDIDIDDDDMMMMMMM
FAM189A1      IDMIIMMDDMMMMMMMM
PDE7A         IDMDDMMIIMMMMMMMM
KIDINS220     IIMIIDIDIDDDDMMMMMMM
UBR2          DDDMMMMMMMMMMMVVV
TUBA1A        DDDMMMMMMMMMMMVVV
SDK1          DDDMMMMMMMMMMMVVV
DLGAP4        DDDMMMMMMMMMMMVVV
HMGB2         DDDMMMMMMMMMMMVVV
P4HA1         DDDMMMMMMMMMMMVVV
YPEL2         DDDMMMMMMMMMMMVVV
FAM13C        DDDMMMMMMMMMVVVVVDD
RASSF3        DDDMMMMMMMMMMMVVV
NTRK2         IDDMMMMMMMMMVVVVDDD
TPT1          IDDMMMMMMMMMMMMI
MTUS1         IDDMMMMMMMMMMMMI
CSNK2B        IDDMMMMMMMMMMMMI
TIMM8B        IDDMMMMMMMMMMMMI
MARCKS        DDDMMMMMMMMMMMIII
DCLK1         IIIMMMMMMWWWWWIIMMM
LRRFIP1       IDDMMMMMMMMMMIMM
LDLRAD4       IDDMMMMMMMMMMIMM
GULP1         IDDMMMMMMMMMMIMM
CARMN         IDDMMMMMMMMMMIMM
BRAF          IDDMMMMMMMMMMIMM
AGO2          IDDMMMMMMMMMMIMM
KMT2E         IDDMMMMMMMMMMIMM
ANK3          IDDMMMMMMMMMMIMM
VMP1          IDDMMMMMMMMMMIMM
ABHD18        IDDMMMMMMMMMMIMM
LINC00310     DDDMMMMVVVVDMMMMMM
ADAMTS7       IDDMMMMMMMMIMMMM
SLC5A3        IDDMMMMMIMMMMMMM
LRRC7         DDDMMMMMIIIMMMMMM
PHLPP1        DDDMMMMMIIIMMMMMM
RPS15         IIIMMMMDDDMMMMMMM
STOX2         IIIMWWWIDIDMMMMMMMM
MRFAP1        IIIMMMMDDDMMMMMMM
NXN           IIIMMMMDDDMMMMMMM
CTNNAL1       IIIMMMMDDDMMMMMMM
NDUFA12       IIDMMMMDIDMMMMMMM
CD99          IIIMMMMDDDMMMMMMM
FRY           IIDMMMMDMMMMMMMM
TBC1D8        IIIMMMMDDDMMMMMMM
FTH1          IIDMMMMDIDMMMMMMM
RPS23         IIIMMMMDDDMMMMMMM
EEF1B2        IIIMMMMDDDMMMMMMM
RPS4X         IIIMMMMDDDMMMMMMM
TTC28         IIIMMMMDDDMMMMMMM
MYOM2         IIIMMMMDDDMMMMMMM
RPL37         IIIMMMMDDDMMMMMMM
ATP5PF        IIDMMMMIDDMMMMMMM
EDF1          IIDMMMMDIDMMMMMMM
CHCHD2        IIDMMMMDIDMMMMMMM
TESC          IIIMMMMDDDMMMMMMM
RPL32         IIIMMMMDDDMMMMMMM
RPS6          IIIMMMMDDDMMMMMMM
RPLP0         IIIMMMMDDIDDMMMMMM
PLXNA4        IIIMMMMIDIDDDDMMMMM
RPS9          IIIMMMMDDDMMMMMMM
RPL24         IIIMMMMDDDMMMMMMM
RPS3          IIIMMMMDDDMMMMMMM
NDUFB9        IIDMMMMIDIDDMMMMMM
RPL7          IIDMMMMIDDMMMMMMM
SYNPR         DDDMMMMIIIMMMMMMM
FAU           IIIMMMMDDDMMMMMMM
MAN1A1        IIIMMMMDDDMMMMMMM
RPS7          IIIMMMMDDDMMMMMMM
UQCR10        IIDMMMMDIDMMMMMMM
MT-ATP6       IIIMMMMDDDMMMMMMM
PREP          IIIMMMMDDDMMMMMMM
RPL13A        IIIMMMMDDDMMMMMMM
COL4A6        IIDMMMMDMMMMMMMM
RPL18A        IIIMMMMDDDMMMMMMM
UNC5C         IDDMMMMIMMMMMMMM
RPL5          IIIMMMMDDDMMMMMMM
ATP1B1        IIDMMMMDIDMMMMMMM
PFDN5         IIIMMMMDDDMMMMMMM
RPL22         IIDMMMMDMMMMMMMM
ATP5MG        IIDMMMMDIDMMMMMMM
CHCHD10       IIDMMMMIDDMMMMMMM
CALM3         IIDMMMMIDDMMMMMMM
SOD1          IIIMMMMDDDMMMMMMM
RPL9          IIIMMMMDDDMMMMMMM
DGKD          IIIMMMMDDDMMMMMMM
C19orf53      IIDMMMMDIDMMMMMMM
NDUFB7        IIDMMMMIDDMMMMMMM
NSD3          IIIMMMDDDMMMMMMMM
MBOAT2        IIIMMMDDDMMMMMMMM
KDELR2        IIIMMMDDDMMMMMMMM
SUB1          IIDMMMIDIDDMMMMMMM
SAFB2         IIIMMMDDDMMMMMMMM
DGLUCY        IIIMMMDDDMMMMMMMM
NUTM2A-AS1    IIDMMMDMMMMMMMMM
C5orf24       IDDMMMDMMMMMMMMVV
OPTN          IIIMMMDDDMMMMMMMM
UBE2E2        IIIMMMDDDMMMMMMMM
PPP2R5C       IDDMMMIIDMMMMMMMM
MB            IIIMMMDDDDDMMMMMMII
TEAD1         IDDMMMIDMMMMMMMIID
RPS2          IIDMMMDMMMMMMMMM
RPL36         IIIMMMDDDMMMMMMMM
MAGED1        IDDMMMIIDIDMMMMMMM
CCN2          IIIMMMDDIDDMMMMMMM
GRAMD4        IIDMMMDMMMMMMMMM
RCAN1         IIIMMMDDDMMMMMMMM
ACOT11        IIIMMMDDDMMMMMMMM
HSPA1B        IIIMMMDDDMMMMMMMM
PALM2-AKAP2   IIDMMMDMMMMMMMMM
PTPRM         IIIMMMDDDMMMMMMMM
LYPLAL1       DDDMMMIIIMMMMMMMM
RGL1          IDDMMMDMMMMMMMMVV
HIPK2         IIIMMMDDDMMMMMMMM
HNRNPC        IDDMMMIDIDIMMMMMMM
RPS15A        IIDMMMDIDMMMMMMMM
YBX1          IIDMMMIDIDDMMMMMMM
COX6C         IDDMMMIIDIDMMMMMMM
COPS9         IIDMMMDIDMMMMMMMM
PTMA          IIDMMMDIDMMMMMMMM
INPP4B        IIIMMMDIDDDMMMMMMM
NDUFV2        IIDMMMIDDMMMMMMMM
RPL23A        IIIMMMDDDMMMMMMMM
ZADH2         IIIMMMDDIDDMMMMMMM
SLK           IDDMMMIMMMMMMMMM
HSPA1A        IIIMMMDDDMMMMMMMM
SKI           IIIMMMDDDMMMMMMMM
POLR2F        IIIMMMDDDMMMMMMMM
HSP90B1       IIIMMMDDDMMMMMMMM
SEPTIN11      IDDMMMIIDMMMMMMMM
SNX3          IIDMMMIDIDDMMMMMMM
RPL7A         IIDMMMDMMMMMMMMM
TSPAN9        IDDMMMIMMMMMMMMM
TNKS          IIIMMMDDDMMMMMMMM
ATP1A1        IIIMMMDDDMMMMMMMM
SNRPC         IIIMMMDIDDDMMMMMMM
TCAP          IDDMMMIIDIDMMMMMMM
OGT           IIIMMMDDDMMMMMMMM
NDUFA6        IDDMMMIDIDIMMMMMMM
BTF3          IIDMMMIIDDDMMMMMMM
NME2          IIDMMMDMMMMMMMMM
N4BP2L2       IIIMMMDDDMMMMMMMM
AC119673.2    IIDMMMDDDMMMMMMMVV
ELOB          IIDMMMDMMMMMMMMM
CIRBP         IDDMMMIIDIIDDMMMMMM
CORO1C        IIIMMMDDDMMMMMMMM
VSNL1         IDDMMMIIDIDMMMMMMM
PPDPF         IDDMMMIIDIDMMMMMMM
FEM1C         IDDMMMIIDMMMMMMMM
MACF1         IDDMMMIMMMMMMMMM
TMSB10        IIIMMMDDDMMMMMMMM
ENO1          IDDMMMIIDIDMMMMMMM
LRRTM3        IDDMMMIIDIIDDMMMMMM
NDUFB3        IIDMMMIDIDDMMMMMMM
ARF1          IDDMMMIDIDIMMMMMMM
HNRNPA3       IIIMMMDDDMMMMMMMM
ENAH          IIIMMMDDDMMMMMMMM
FAM13B        IIIMMMDDDMMMMMMMM
PAG1          IIIMMMIIDIDDDMMMMIDDD
SIVA1         IIDMMMDMMMMMMMMM
KRAS          IIIMMMDDIDDMMMMMMM
RRAD          IIDMMMDMMMMMMMMM
ZNF521        IIDMMMDMMMMMMMMM
LCLAT1        IDDMMIDIMMMMMMMMM
GREB1L        IDDMMIDIDIMMMMMMMM
LMO4          IDDMMDDMMMMMMMMVVV
DHRS3         IIIMMDIDIDDDMMMMMMM
PABPC1        IDDMMDMMMMMMMMMVV
SNRPD2        IDDMMDMMMMMMMMMVV
HECW2         IIIMMDDDDDDMMIIIMMMM
SMAD7         IDDMMDMMMMMMMMMII
TGFBR3        IDDMMIIDIDMMMMMMMM
GUK1          IIIMMDIDIDDDMMMMMMM
ARMCX3        IIIMMDDIDDMMMMMMMM
PITPNC1       IIIMMDDDDDMMMMMMMVV
ATP6V0C       IDDMMIMMMMMMMMMM
CD63          DDDMMIDIDIIIIDMMMMMM
KCNJ3         IDDMMIDIDIMMMMMMMM
TSPAN18       IIIMMDIDDDMMMMMMMM
ZFHX3         IIDMMIDIDDMMMMMMMM
SLC4A4        IDDMMIDIDIMMMMMMMM
ZNF608        IDDMMIDIDIIIDDMMMMMM
MBTPS1        IDDMMIDIMMMMMMMMM
BAMBI         IDDMMIIDMMMMMMMMM
AP1S2         IIIMMDIDIDDDMMMMMMM
CYP2J2        DDDMMIIIMMMMMMMMM
SRL           IDDMMIDIDIMMMMMMMM
AC083870.1    IIIMMIIDDDDDMMMMMMM
CADPS2        IDDMMIIDIDMMMMMMMM
SPATS2L       IDDMMIDMMMMMMMIMM
GRIP1         IDDMMIDIDIMMMMMMMM
NONO          IDDMMDDMMMMMMMMVVV
RBFOX1        IIIMMDDDDDDMMMMMMVVV
ZFAND5        IDDMMIDIMMMMMMMMM
RGS6          IDDMMDIDMMMMMMMMII
YWHAZ         IDDMMIMMMMMMMMMM
MAP1B         IDDMMDDMMIIIMMMMMM
PLN           IIIMMDDIDDMMMMMMMM
HMGA2         IDDMMIIDMMMMMMMMM
NDRG4         IIIMMDIDIDDDMMMMMMM
TMEM71        DDDMMDDMMMIIIMMMMII
INSR          IDDMMDDMMMMMMMMVVV
SYNPO2        IIDMMIDIDIDDMMMMMMM
NFIA          IDDMMIDMMMMMMMIMM
RABGEF1       IIIMMDDDMMMMMMMMM
EPHA4         IIIMMDDDDMMIMMMMMM
BHLHE40       IDDMMIDIDIMMMMMMMM
NDUFA11       IDDMMIDIDIIDMMMMMMM
PPP2R5A       IIIMMDDDMMMMMMMMM
GJC1          IDDMMDMMMMMMMMMVV
SPHKAP        DDDMIIDIDIIIDMMMMMMM
GLI3          DDDMIIIDIMMMMMMMMM
CDH11         IDDMIDIDIIDMMMMMMMM
SFRP1         IIDMIIDIDIIDDDDMMMMMM
SOBP          IDDMIIDIDIIDDMMMMMMM
C1orf21       IDDMIIDIDIIDDMMMMMMM
SMAD6         IDDMIIDIDIIDDMMMMMMM
CHRM2         DDDMIIDIDIIIDMMMMMMM
FAT3          IDDMIIIMMIDDDMMMMMM
SLC2A3        IIDMIDIDIDDMMMMMMMM
FAM124A       DDDMIDMMMMMMMMMVVV
TACC2         DDDMDDMMMIIIIIMMMMM
RFC3          DDIDMMMMMMMMMMMVV
EXOC6B        DDIDMMMMMMMMMMMVV
GLCE          DDIDMMMMMMMMMMMVV
PTAR1         DDIDMMMMMMMMMMMVV
NPTN          DDIDMMMMMMMMMMMVV
RB1CC1        DDIDMMMMMMMMMMMVV
VIM           DDIDMMMMMMMMMMMVV
RPS25         IDIDMMMMMMMMMMMM
TLE5          IDIDMMMMMMMMMMMM
HEG1          IDIDMMMMMMMMMMMM
ZFPM2         IDIDMMMMMMMMMMMM
SNHG29        IDIDMMMMMMMMMMMM
MIR99AHG      IDIDMMMMMMMMMMMM
NFIB          IDIDMMMMMMMMMMMM
PLAAT3        IDIDMMMMMMMMMMMM
AC092691.1    IDIDMMMMMMMWWWWWIIIII
ARL6IP4       IDIDMMMMMMMMMMMM
CDKN1A        IDIDMMMMMMMMMMMM
MXRA7         IDIDMMMMMMMMMMMM
ATP1B2        IDIDMMMMMMMMMMMM
ESRRG         IDIDMMMMMMMMMMMM
TENM4         IDIDMMMMMMMMMMMM
RRAGD         IDIDMMMMMMMMMMMM
PAPOLA        IDIDMMMMMMMMMMMM
ZEB2          IDIDMMMMMMMMMMMM
RPS12         IDIDMMMMMMMMMMMM
RAI1          IDIDMMMMMMMMMMMM
RIPK4         IIIDMMMMMMMMMMDDM
PDE1A         IIIDMMMMMMWWWWIIMMM
LRRC4C        IDIDMMMMMMMMIIDIIDDD
FITM1         DDDDMMMMMIIIIMMMMM
ERC2          DDDDMMMMMIIIIMMMMM
SLC8A3        IDIDMMMMIDMMMMMMID
BNIP3         DDDDMMMMIIIIMMMMMM
TOMM7         IIIDMMMMDDMMMMMMM
RPL8          IIIDMMMMDDMMMMMMM
ST3GAL4       IIIIMMMMDDDDMMMMMM
MDH2          DDDDMMMMIIIIMMMMMM
NR3C2         IIIDMMMMDDMMMMMMM
CYTH3         DDDDMMMMIIIIMMMMMM
AC087457.1    DDIDMMMMIIMMMMMMM
TNNI1         IIIDMMMMDDMMMMMMM
TOMM20        IIIDMMMMDDMMMMMMM
ALPK2         DDIDMMMMIIMMMMMMM
RALGPS2       DDDDMMMMIIIIMMMMMM
LRMDA         DDDDMMMMIIIIMMMMMM
FBXO32        IIIIMMMMDDDDMMMMMM
ADGRL2        DDDDMMMMIIIIMMMMMM
NDUFA4        IIIDMMMMDDMMMMMMM
COX7C         IIIDMMMMDDMMMMMMM
KDM6A         IIIDMMMDDMMMMMMMM
CPNE3         IIIDMMMDDMMMMMMMM
RPS20         IIIDMMMDDMMMMMMMM
MT-CYB        IIIIMMMDDDDMMMMMMM
BMP7          IIIIMMMDDDDMMMMMMM
SAMD12        IIIIMMMDDDDMMMMMMM
ATP5PD        IDIDMMMIIDDMMMMMMM
DAB2IP        IIIIMMMDDDDMMMMMMM
MEF2A         IIIDMMMDDMMMMMMMM
JUND          IIIDMMMDDMMMMMMMM
COX5A         IDIDMMMIDIIDDMMMMMM
SRGAP1        IIIIMMMDDDDMMMMMMM
MAGI1         IIIDMMMDIDDMMMMMMM
NDUFB1        IIIDMMMIDDDMMMMMMM
FBRSL1        IIIDMMMDDMMMMMMMM
RPL35         IIIDMMMDDMMMMMMMM
RPL35A        IIIIMMMDDDDMMMMMMM
RPS27         IIIDMMMDIDDMMMMMMM
BACH2         IIIDMMMDDMMMMMMMM
PHPT1         IIIDMMMDIDDMMMMMMM
ITGB1         IIIDMMMDDMMMMMMMM
RPL37A        IIIIMMMDDDDMMMMMMM
SLC1A3        DDDDMMMIIIIMMMMMMM
SMPX          IIIIMMMDDDDMMMMMMM
EIF3H         DDDDMMMIIIIMMMMMMM
TRDN-AS1      IIIDMMMDDMMMMMMMM
AZIN1         IIIDMMMDDMMMMMMMM
GJA5          IIIIMMMDDIDDDMMMMMM
PLCXD3        DDDDMMMIIIMMMMMIMM
HK1           IIIDMMMIDDDMMMMMMM
RPL36AL       IIIDMMMDDMMMMMMMM
FIS1          IIIDMMMDIDDMMMMMMM
EEF1A1        IIIIMMMDDDDMMMMMMM
PPP1R3C       IIIDMMMDDMMMMMMMM
COPE          IIIDMMMDIDDMMMMMMM
RBM6          IIIIMMMDDDDMMMMMMM
EFNA5         IIIIMMMDDDDMMMMMMM
ATP5F1B       IIIDMMMIDIDDDMMMMMM
NDUFB10       IDIDMMMIIDDMMMMMMM
NDUFAF8       IIIDMMMDDMMMMMMMM
FLNC          IIIDMMMDIDDMMMMMMM
RPL10         IIIDMMMDIDDMMMMMMM
SORT1         IIIIMMMDDDDMMMMMMM
COX8A         IIIDMMMIDIDDDMMMMMM
RPS3A         IIIIMMMDDDDMMMMMMM
RPS17         IIIDMMMDDMMMMMMMM
OBSCN         IIIIMMMDDDDMMMMMMM
RPS16         IIIDMMMDDMMMMMMMM
COX7B         IDIDMMMIIDDMMMMMMM
RPL13         IIIDMMMDDMMMMMMMM
NDUFS6        IDIDMMMIDIDMMMMMMM
COX6B1        IDIDMMMIIDDMMMMMMM
MT-CO2        IIIIMMMDDDDMMMMMMM
DUOX2         IIIIMMMDDIDDDMMMMMM
RPS21         IIIIMMMDDDDMMMMMMM
RPL30         IIIIMMMDDDDMMMMMMM
CSRP3         IIIIMMMDDDDMMMMMMM
ATXN1         IIIIMMMDDDDMMMMMMM
APLP2         IIIDMMMDDIDMMMMMMM
ABCB4         IIIIMMMDDDDMMMMMMM
GAPDH         IIIDMMMIDDDMMMMMMM
PTK2          IDIDMMIDMMMMMMMMM
SLC2A1        DDDDMMIIIIMMMMMMMM
RFTN1         IDIDMMIIDIIIIIDDDDDMMMD
CMSS1         IDIDMMIDIDMMMMMMMM
NRP2          DDIDMMIIMMMMMMMMM
LDB3          IDIDMMIIDIDDMMMMMMM
HAND2-AS1     IIIIMMDDDDMMMMMMMM
RABAC1        IDIDMMIIDIDDMMMMMMM
BARD1         IIIDMMDIDDMMMMMMMM
CGNL1         IDIDMMDDMMMMMMMMVV
PPP2R1A       IIIIMMDIDDDDMMMMMMM
IRF4          IIIIMMDDDDDDMMMMMMVV
STARD9        IDIDMMDDMMMMMMMMVV
DNAJC15       IDIDMMIDMMMMMMMMM
RSRC1         IDIDMMIIDDMMMMMMMM
FOXO3         IDIDMMDDMMMMMMMMVV
RCSD1         IDIDMMIDMMMMMMMMM
FGF1          IIIIMMDDDDMMMMMMMM
SGCD          DDDDMMIIMMMMMMMMVV
PRICKLE1      IDIDMMIDIDDMMMMMMIID
CTIF          IIIIMMDDDDMMMMMMMM
KRT10         IIIIMMDIDDDDMMMMMMM
PLD1          IDIDMMIIDDMMMMMMMM
SUPT3H        IDIDMMIIDDMMMMMMMM
STAG2         IIIIMMDDDDMMMMMMMM
PGM5          IDIDMMDDMMMMMMMMVV
TANC2         IDIDMMDDMMMMMMMMII
CDH4          IIIIMMIDIDIIDDDDDDMMMM
SELENOT       IIIDMMIDIDDDMMMMMMM
HNRNPA0       IDIDMMDDMMMMMMMMVV
PTPRK         IDIDMMIDIDMMMMMMMM
MYL12A        IIIDMMDDMMMMMMMMM
NEXN          DDDDMMIIMMMMMMMMVV
CLIP4         IDIDMMIIDIDDMMMMMMM
CHST11        IDIDMIIDIIDDDDDMMMMIIID
NREP          IIIDMDDIDMMMMMMMMM
TSC22D1       DDIDMDIDMMMMMMMMVVV
MYL3          DDDDMIIDIIMMMMMMMMI
TRABD2B       IDIDMDDDMIIIMMMMMMM
TMEM178A      DDIDMIIDIIDMMMMMMMM
FREM2         DDIDMIIMMMMMMMMMM
FBXO40        DDIDMIDIDMMMMMMMMVV
CDK14         IDIDMDIDMMMMMMMIMM
WSB1          DDIDMIDIDIIMMMMMMMM
MID1          IIIDMIDIDDDMMMMMMMM
RBPMS2        DDDDMIIIIMMMMMMMMM
AC008591.1    DDIDMDMMMMMMMMMVVV
DES           DDIDMDDMIIIIMMMMMMM
XYLT1         DDDDMIIIIIDMMMMMMMM
CAST          DDIDMDMMMMMMMMMVVV
TMEM163       DDDDMIIDMMMMMMMMIII
PRDM6         IDIDMIIDIIDDDMMMMMMM
ENOX1         IDIDMIMIIDDDMMMMMMM
ERO1A         DDDIDMMMMMMMMMMVVV
STK38L        DDDIDMMMMMMMMMMVVV
CREB5         DDDIDMMMMMMMMMMVVV
RPS10         DDDIDMMMMMMMMMMVVV
NKX2-5        DDDIDMMMMMMMMMMVVV
FOXN3         DDDIDMMMMMMMMMMVVV
ELF1          DDDIDMMMMMMMMMMVVV
MPPED2        DDDIDMMMMMMMMMMVVV
MLYCD         DDDIDMMMMMMMMMMVVV
NBEA          DDDIDMMMMMMMMMMVVV
CAMK1D        DDDIDMMMMMMMMMMIII
MYO16         IIDIDMMMMMMMMMMIDD
CASQ2         DDDIDMMMMMMMMMMIII
PCBP2         IDDIDMMMMMMMMMIMM
RGS3          IDDIDMMMMMMMMMIMM
NEURL1B       DDDIDMMMMVVVVDMMMMM
RPL39         DDDDDMMVVVVIMMMMMMM
ATP5IF1       IIIIDWWWWWWIIIMMMMMMM
TPX2          IIIDIMMMMMDDDMMMMM
TUBA1B        DDDDIVVVVVDDMMMMMMMM
FMNL2         IDDIDMMMMMIMMMMMM
ULK4          DDDDDMMMMIIIIIMMMMM
PPP2R2B       DDDIDMMMMIIIMMMMMM
MYPN          DDDDDMMMMIIIIIMMMMM
MYH7B         IDDIDMMMIMMMMMMMM
RPL4          IIIDIMMMDDDMMMMMMM
EGFR          DDDDDMMMIIIIIMMMMMM
BNC2          DDDDDMMMIIIIIIIIIIIDWWWWW
KIF26B        IIIIIMMMIIDDDDMMMIDDDD
ATP13A3       DDDIDMMMIIIMMMMMMM
PGAM2         DDDDDMMMIIIIIMMMMMM
PRNP          IIIDIMMMDDDMMMMMMM
DAPK2         IIIDIMMMDDMMMMMMMD
RPS18         IIIDIMMMDDDMMMMMMM
SERF2         IIIDIMMMDDDMMMMMMM
SGIP1         IIIIIMMMDDDDDMMMMMM
RPLP1         IIIDIMMMDDDMMMMMMM
RPL23         IIIDIMMMDDDMMMMMMM
DAB1          IIIIIMMMIIIDDDDDDDMMMD
BRI3          IIDIDMMMIDDMMMMMMM
CMC2          DDDDDMMMIIIIIMMMMMM
SLC27A6       IIIDIMMMDDDMMMMMMM
RPS26         IIIDIMMMDDDMMMMMMM
MICOS10       IIIDIMMMDDDMMMMMMM
MIF           IIDIDMMMDMMMMMMMM
NDUFS5        IIDIDMMMIDDMMMMMMM
PRKCE         IIIIIMMMDDDDMMMMMMD
TNNC1         IIIDIMMMIDDDDMMMMMM
HSPB2         IIDIDMMMIDDMMMMMMM
ATP5MPL       IIIDIMMMDDDMMMMMMM
GOLIM4        DDDIDMMMIIIMMMMMMM
NDUFB11       DDDDDMMMIIIIIMMMMMM
MBP           IIIIIMMMDDDDDMMMIIIDDD
COX5B         IIIDIMMMDDDMMMMMMM
RPS8          IIIDIMMMDDDMMMMMMM
GAS5          IIIDIMMDDDMMMMMMMM
ADGRL3        IDDIDMMDMMMMMMMMVV
NDUFA1        DDDDDMMIIIIIMMMMMMM
MT-CO3        IIIIIMMDDDDDMMMMMMM
FHL1          IIIIIMMDDDDDMMMMMMM
AC011900.1    DDDDDMMIIIIIMMMMMMM
ATP5F1E       IIDIDMMIDIDDMMMMMMM
PLCL2         DDDIDMMIIIMMMMMMMM
JAK1          IIIDIMMDDDMMMMMMMM
MBNL1         IIIDIMMDDDMMMMMMMM
PEAK1         IIIIIMMDDDDDMMMMMMM
BEX1          DDDDDMMIIIIIMMMMMMM
UQCRQ         IIIIIMMDDDDDMMMMMMM
RPS13         IIDIDMMDMMMMMMMMM
EEF2          IIDIDMMDMMMMMMMMM
UBE2E3        IIDIDMMDMMMMMMMMM
RAB21         IIIIIMMDDDDDMMMMMMM
CDC42EP3      IIDIDMMDMMMMMMMMM
NR2F1-AS1     IIIDIMMDDDMMMMMMMM
KHDRBS2       IDDIDMMDMMMMMMMMVV
DSP           DDDDDMMIIIIIMMMMMMM
PDGFC         DDDIDMMIIIIIDDMMMMMM
RPL12         IIIDIMMDDDMMMMMMMM
DNAH11        DDDDDMMIIIMMMMMMMVV
ATP5MF        IIDIDMMIIDDDMMMMMMM
FNDC3B        IIIDIMMDDDMMMMMMMM
GNG5          IIIDIMMDDDMMMMMMMM
MAGI2         DDDIDMMIIDIIMMMMMMM
RNASEK        IIDIDMMIDIDDMMMMMMM
RPL29         IIIDIMMDDDMMMMMMMM
FREM1         IIIDIMMDDDMMMMMMMM
MYOZ2         IIIIIMMDDDDDMMMMMMM
CKM           DDDIDMIDIIIIDMMMMMMM
HSDL1         DDDIDMIIIMMMMMMMMM
JUN           IIIIIMDDDDDMMMMMMMM
NMNAT2        IIIIIMDDDDDMMMMMMMM
TANC1         DDDDDMIIDIIIIIIDDMMMMM
FOXP2         DDIDIDMMMMMMMMMMVV
LAPTM4A       DDIDIDMMMMMMMMMMVV
CDH13         DDIDIDMMMMMMMMMMVV
PRKN          DDIDIDMMMMMMMMMMVV
FBXO31        DDIDIDMMMMMMMMMMVV
SFRP5         IDIDIDMMMMMMMMMMM
DSC3          IDIDIDMMMMMMMMMMM
KANK1         IDIDIDMMMMMMMMMMM
TNS1          IDIDIDMMMMMMMMMMM
SPATA5        IDIDIDMMMMMMMMMMM
FZD1          IDIDIDMMMMMMMMMMM
GDF10         IIIDIDMMMMMMMMMMDD
AC008050.1    DDIDIDMMMMMMMMMMII
E2F8          DDIDIDMMMMMMMMIIMM
RAP1GDS1      DDIDIDMMMMIIMMMMMM
LMOD2         DDDIDDMMMMIIIIMMMMM
FBXO42        DDDIDDMMMMIIIIMMMMM
MED12L        DDDIDDMMMMIIIIMMMMM
B2M           IIDIDIVVVVDDDDDDMMMMMM
ELL2          DDDIDDMMMIIIIMMMMMM
RREB1         DDDIDDMMMIIIIMMMMMM
UQCR11        DDDIDDMMMIIIIMMMMMM
RPL11         DDDIDDMMMIIIIMMMMMM
AC018464.1    IIIDIDMMDDMMMMMMMM
DLC1          IIIDIDMMDDMMMMMMMM
IGDCC4        IIIIIIMMDDDDDDMMMMMM
COX7A1        IIIDIIMMDDDDMMMMMMM
CACNA1C       DDDDDDMMIIIMMMMMMVVV
DIAPH3        IIIIIIMMDDDDDDMMMMMM
LRRC49        IIIDIDMMDDMMMMMMMM
PYGB          DDIDIDMMIIMMMMMMMM
COX6A2        IIIDIDMMIDDDMMMMMMM
COL4A1        IIIIIIMMDDDDDDMMMMMM
MIR924HG      DDDDDDMMIIIIIIMMMMMM
RASGEF1B      IIIDIIMMDDDDMMMMMMM
ADAMTS8       IIIIIIMMDDDDDDMMMMMM
NDUFAB1       IIIIIIMMDDDDDDMMMMMM
LRRFIP2       DDDIDDMMIIIIMMMMMMM
FAT1          IIIIIIMMDDDDDDMMMMMM
FABP3         IIIDIDMMIDDDMMMMMMM
PDE4DIP       IDIDIDMMIDMMMMMMMM
FAM189A2      IIIIIIMMDDDDDDMMMMMM
FARP1         IIIIIIMMDDDDDDMMMMMM
PRSS35        IIIDIDMMDDMMMMMMMM
TCF7L1        IIIDIDMMDDMMMMMMMM
LAMA4         IIIDIDMMDDMMMMMMMM
ID2           IIIDIDMMDDMMMMMMMM
CPVL          DDDIDDMIIIIMMMMMMMM
EIF1B         DDDIDDMIIIIMMMMMMMM
HIF1A-AS3     DDDIDIDMMMMMMMMMVVV
PRKAA2        DDDIDIDMMMMMMMMMVVV
DACH2         DDDIDIDMMMMMMMMMVVV
CYP7B1        DDDIDIDMMMMMMMMMVVV
GPRIN3        DDDIDIDMMMMMMMMMVVV
MRPL41        DDDIDIDMMMMMMMMMVVV
RNF216        DDDIDIDMMMMMMMMMVVV
AC073050.1    IIIIIIIMMMMMMMWWWWWDD
INTS6L        DDDIDIDMMMMMMMMMVVV
MITF          DDDIDIDMMMMMMMMMVVV
CASC15        IIDIDIDMMMMMMMMDMM
TXLNB         IDDIDIDMMMMMMMMIMM
PLPP7         IDDIDIDMMMMMMMMIMM
DLG2          DDDIDDDMMVVVIIMMMMMM
COL13A1       DDDIDDDMMMMIIIIIMMMM
RCAN2         DDDIDIDMMMIIIMMMMMM
ABRA          IIIIIDIMMDDDDDMMMMMM
ANGPT1        IIIDIDIMMIIIDDDDDMMMMD
ARHGAP24      IIIIIDIMMDDDDDMMMMMM
ARHGAP17      DDDIDIDMMIIIMMMMMMM
CRIP2         DDDIDDDMMIIIIIMMMMMM
MRPS6         DDDIDDDMMIIIIIMMMMMM
ITPR2         IIIIIDIMMDDDDDMMMMMM
LMTK2         DDDIDDDMIIIIIMMMMMMM
AC009878.2    DDDIDDDMIIIIIIDMMMMMM
ITGA6         IIIDIDIMDDDMMMMMMMM
NEDD4L        DDDIDDDMIIIIIIDMMMMMM
EPB41L3       IIIIIDIMDDDDDMMMMMMM
THSD4         DDDIDDDMIIIIIIIDDMMMMM
AC147651.1    DDDIDDDMIIIIIMMMMMMM
NACA          DDIDIDIDMMMMMMMMMVV
NRXN1         DDIDIDIDMMMMMMMMMVV
NEGR1         IIIDIDIDMMMMMMMMMWW
TTC7B         DDIDIDIDMMMMMMMMMVV
OSBPL2        DDIDIDIDMMMMMMMMMVV
LTBP1         DDIDIDIDMMMMMMMMMVV
CAMK2G        IIIDIDIDMMMMMMMMMWW
KANSL1        DDIDIDIDMMMMMMMMMVV
CPEB3         IDIDIDIDMMMMMMMMMM
B4GALNT3      IDIDIDIDMMMMMMMMMM
OXCT1         IDIDIDIDMMMMMMMMMM
ZMIZ1         IDIDIDIDMMMMMMMMMM
RPL27A        DDIDIDIDMMMMMMMMMII
RPS24         DDIDIDIDMMMMMMMMMII
APBA1         DDIDIDIDMMMMMMMMMII
TGFB2         IIIDIDIDMMMMMMMMDDM
VDAC3         DDDDDDDIVVVVVIMMMMMMM
RPS11         DDDDDDDIVVVVVIMMMMMMM
ADARB2        DDIDIDIDMMMMMIIMMMM
CLSTN2        IIIDIDIDMMMMDDMMMMM
PRDX1         DDDIDDDDMMIIIIMMMMIIID
GRK5          DDDIDIDDMMIIIIMMMMMM
MYLK3         IIIDIDIDMMDDMMMMMMM
ATP5MC2       DDDIDIDDMMIIIIMMMMMM
CXCL12        DDDIDIDDMMIIIIMMMMMM
PLCB1         IIIIIDIIMDDDDDDMMMMMID
ROR2          DDDIDIDDMIIIIMMMMMMM
MMP15         DDDIDIDIDMMMMMMMMVVV
BANF1         DDDIDIDIDMMMMMMMMVVV
PAPPA         IIDIDIDIDMMMMMWWWWWIIII
UQCRB         DDDIDIDIDMMMMMMMMIII
SLC24A3       IIDIDIDIDMMMMMMMMDM
FYN           IDDIDIDIDMMMMMMMIMM
PPP1R3B       DDIDIDIDIDMMMMMMMMVV
CDK18         DDIDIDIDIDMMMMMMMMVV
ADAMTS9-AS2   DDIDIDIDIDMMMMMMMMVV
WDR60         DDIDIDIDIDMMMMMMMMVV
IDH3A         IDIDIDIDIDMMMMMMMMM
NID1          IDIDIDIDIDMMMMMMMMM
GPX3          IDIDIDIDIDMMMMMMMMM
VASH1         IDIDIDIDIDMMMMMMMMM
AEBP2         IDIDIDIDIDMMMMMMMMM
BMT2          IDIDIDIDIDMMMMMMMMM
VCAN          IDIDIDIDIDMMMMMMMMM
DYRK1A        IDIDIDIDIDMMMMMMMMM
RPL38         IDIDIDIDIDMMMMMMMMM
RBMS3         DDIDIDIDIDMMMMMMMMII
EMC10         DDIDIDIDIDMMMMMMMMII
SH3D19        DDIDIDIDIDMMMMMMMMII
EPS8          DDIDIDIDIDMMMMMMMIIID
FNIP2         DDIDIDIDIDMMMMMMIIMM
RICTOR        DDIDIDIDIDMMMMMMIIMM
MIR100HG      IIIIIIIIIIMMIIDDDDDDDDDDDD
KCNMB2        IIIIIIIIIIMMIIDDDDDDDDDDDD
ZNF385B       IIIIIIIIIIMMIIDDDDDDDDDDDD
FAM155A       DDIDIDIDIDDMMMMMMMVVV
PATJ          IDIDIDIDIDDMMMMMMIMM
L3MBTL4       IDIDIDIDIDDDMMMMMMMVV
CAV1          IIIDIDIDIDDDMMMMMMMM
CAV2          IIIDIDIDIDDDMMMMMMMM
SIK3          IIIDIDIDIDDDMMMMMMMM
AC020909.2    IIIDIDIDIDDDMMMMMMMM
NAMPT         IIIDIDIDIDDDMMMMMMMM
TMEM108       IIIDIDIDIDDDMMMMMMMM
WNK1          IIIDIDIDIDDDMMMMMMMM
ANK2          IIIDIDIDIDDDMMMMMMMM
PDLIM1        IIIDIDIDIDDDMMMMMMMM
HOMER1        IIIDIDIDIDDDMMMMMMMM
AC109466.1    IIIDIDIDIDDDMMMMMMMM
AL078590.2    IIIDIDIDIDDDMMMMMMMM
SETBP1        IIIDIDIDIDDDMMMMMMMM
NAV2          IIIDIDIDIDDDMMMMMMMM
CRYAB         IIIDIDIDIDDDMMMMMMMM
RAPGEF2       IIIDIDIDIDDDMMMMMMMM
ATAD1         IIIDIDIDIDDDMMMMMMMM
PCMTD2        IIIDIDIDIDDDMMMMMMMM
INPP5A        IIIDIDIDIDDDMMMMMMMM
PRELID2       IIIDIDIDIDDDMMMMMMMM
PICALM        IIIDIDIDIDDDMMMMMMMM
ACTN2         IIIDIDIDIDDDMMMMMMMM
CTNNA1        IIIDIDIDIDDDMMMMMMMM
DDR2          IDIDIDIDIDDDMMMMMIIMM
CMYA5         IDIDIDIDIDDDDMMMMMMIII
Z85996.3      IIIDIDIDIDIDDDMMMMMMM
SGSM1         IIIDIDIDIDIDDDMMMMMMM
CPNE4         IIIDIDIDIDIDDDMMMMMMM
KANSL1L       IIIDIDIDIDIDDDMMMMMMM
LINC01184     IIIDIDIDIDIDDDMMMMMMM
MCEMP1        IIIDIDIDIDIDDDMMMMMMM
ERVMER61-1    IIIDIDIDIDIDDDMMMMMMM
THSD7A        IIIDIDIDIDIDDDMMMMMMM
COL1A2        IIIDIDIDIDIDDDMMMMMMM
NTN4          IIIDIDIDIDIDDDMMMMMMM
ADAM19        IIIDIDIDIDIDDDMMMMMMM
RNF150        IIIDIDIDIDIDDDMMMMMMM
LINC01374     IIIDIDIDIDIDDDMMMMMMM
CCSER1        IIIDIDIDIDIDDDMMMMMMM
RPL34         IIIDIDIDIDIDDDMMMMMMM
MLLT10        IIIDIDIDIDIDDDMMMMMMM
RAPGEF4       IIIDIDIDIDIDDDMMMMMMM
TENM3         IIIDIDIDIDIDDDMMMMMMM
LIMCH1        IIIDIDIDIDIIDDDDMMMMMM
WDFY3         IIIDIDIDIDIIDDDDMMMMMM
CTNNB1        IIIDIDIDIDIIDDDDMMMMMM
CKB           IIIDIDIDIDIIDDDDMMMMMM
SH3KBP1       IIIDIDIDIDIIDDDDMMMMMM
EPHA7         IIIDIDIDIDIIDDDDMMMMMM
SLCO3A1       IIIDIDIDIDIIDDDDMMMMMM
RASAL2        IIIDIDIDIDIIDDDDMMMMMM
DUOXA2        IIIDIDIDIDIIDDDDMMMMMM
OTOGL         IIIDIDIDIDIIDDDDMMMMMM
MYO1D         IIIDIDIDIDIIDDDDMMMMMM
HTR4          IIIDIDIDIDIIDDDDMMMMMM
DPYD          IIIDIDIDIDIIDDDDMMMMMM
KIF13A        IIIDIDIDIDIIDDDDMMMMMM
PRRX1         IIIDIDIDIDIIDDDDMMMMMM
CACNB2        IIIDIDIDIDIIDDDDMMMMMID
INHBA         IIIDIDIDIDIIDDDDMMMIIIDDD
UST           IIIDIDIDIDIIIDDDDMMMMMD
DOK6          IIIDIDIDIDIIIDDDDDMMMMM
RABGAP1L      IIIDIDIDIDIIDDDDDDMMMIIID
DGKH          IIIDIDIDIDIIDDDDDDMMMIIID
PDE1C         IIIDIDIDIDIIIIDDDDDDMMMID
```

Example 116 (unknown):
```unknown
102        PITX2
524       DCBLD2
839        PANCR
1089       PCDH7
1254        NAV3
          ...   
573          UST
9           DOK6
539     RABGAP1L
1267        DGKH
808        PDE1C
Name: genes, Length: 1546, dtype: object
```

Example 117 (unknown):
```unknown
ClusterUtils.run_clustering(aligner, metric='levenshtein', DIST_THRESHOLD=0.4)
```

Example 118 (unknown):
```unknown
compute distance matrix
using levenshtein distance metric
run agglomerative clustering |  0.4
silhouette_score:  0.1241659753131663
```

Example 119 (unknown):
```unknown
ClusterUtils.visualise_clusters(aligner,n_cols = 4, figsize= (10,6))
```

Example 120 (unknown):
```unknown
ClusterUtils.print_cluster_average_alignments(aligner)
```

Example 121 (unknown):
```unknown
cluster:  0 IIDDMMMMMMMMMVVVDDD 7
cluster:  1 IIIIMMMMIDDDDMMMMMD 6
cluster:  2 IIIDMMMDDMMMMMMMM 1091
cluster:  3 IIIDIDIDIDIIIIIDDDDDDMMMD 15
cluster:  4 IIDDDMMMMMMMIMMMVD 15
cluster:  5 MIIIMMMMMMMMMMWWW 3
cluster:  6 IIDDDDDMMMIIIMMMMMM 89
cluster:  7 IIDMMMMMMVVVVVDDMWWWW 3
cluster:  8 MIIMVVVDDDDDMMMMMMM 24
cluster:  9 IIIDDDMWWWIIIMMMMMMM 4
cluster:  10 IIIDDDDDDMVMVMVMMMMM 5
cluster:  11 IIDDMMMMMVVDDMMMMM 6
cluster:  12 IDDDDDMWWIIIIIIIIIIIIDDDDDD 1
cluster:  13 IIIDIDIDIDIIIIIIIIDDDDDDDDDD 13
cluster:  14 MMMMMMMMMWWWIIIMM 7
cluster:  15 MMMMWWWWWDMMMIDIIIIII 1
cluster:  16 IIIDIDIDMWMMMMWWWWWIIII 1
cluster:  17 MIMMMMMMMMMMMWM 2
cluster:  18 MMIIIDDDDDMMMMMVMVVD 2
cluster:  19 MMMMMMMMVVVVVVDDDDDD 4
cluster:  20 IIMMIDDDDDMMMIMMMVM 7
cluster:  21 MMMMMMMMVIIMMMDDD 1
cluster:  22 IDMMMIDDMMMMMMMMV 237
cluster:  23 DDDDDMMMIIIIIIIIIIDDDMWW 1
cluster:  24 IDDMIIMVMIDDDMMMMMM 1
```

Example 122 (unknown):
```unknown
# Initialize an empty dictionary
cluster_genes_dict = {}

# Loop through each gene and its corresponding cluster ID
for gene, cluster_id in zip(aligner.gene_list, aligner.cluster_ids):
    # Check if the cluster ID is already in the dictionary
    if cluster_id not in cluster_genes_dict:
        # If not, create a new list for this cluster ID
        cluster_genes_dict[cluster_id] = []
    # Append the gene to the list for this cluster ID
    cluster_genes_dict[cluster_id].append(gene)
```

Example 123 (unknown):
```unknown
clus_of_interest=13
```

Example 124 (python):
```python
cluster_genes_dict[clus_of_interest]
print(len(cluster_genes_dict[clus_of_interest]))
```

Example 125 (unknown):
```unknown
GENE_SUBSET = cluster_genes_dict[clus_of_interest]
aligner.get_aggregate_alignment_for_subset(GENE_SUBSET)
```

Example 126 (unknown):
```unknown
Average Alignment:  IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 127 (python):
```python
for gene in GENE_SUBSET:
    if len(GENE_SUBSET)<150:
        print(gene, aligner.results_map[gene].alignment_str)
        VisualUtils.plotTimeSeries(gene, aligner, vs)
        plt.show()
```

Example 128 (unknown):
```unknown
PITX2 IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 129 (unknown):
```unknown
DCBLD2 IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 130 (unknown):
```unknown
PLCL1 IIMMIDIDIIIIIIIIDDDDDDDDDD
```

Example 131 (unknown):
```unknown
PANCR IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 132 (unknown):
```unknown
MIR100HG IIIIIIIIIIMMIIDDDDDDDDDDDD
```

Example 133 (unknown):
```unknown
KCNMB2 IIIIIIIIIIMMIIDDDDDDDDDDDD
```

Example 134 (unknown):
```unknown
KCNMB2-AS1 MIIIIDIDIDIIIIIIIDDDDDDDDDD
```

Example 135 (unknown):
```unknown
PCDH7 IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 136 (unknown):
```unknown
ZNF385B IIIIIIIIIIMMIIDDDDDDDDDDDD
```

Example 137 (unknown):
```unknown
NAV3 IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 138 (unknown):
```unknown
VASH2 IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 139 (unknown):
```unknown
NTM IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 140 (unknown):
```unknown
ROR1 IIIDIDIDIDIIIIIIIIDDDDDDDDDD
```

Example 141 (python):
```python
import pandas as pd

gene_list = aligner.gene_list
cluster_ids = aligner.cluster_ids

# Convert them into a pandas DataFrame
cluster_df = pd.DataFrame({
    'gene': gene_list,
    'cluster_id': cluster_ids
})


LEFT_genes_set = set(gene_lists['LEFT_genes'])
RIGHT_genes_set = set(gene_lists['RIGHT_genes'])
COMMON_genes_set = set(gene_lists['COMMON_genes'])

# Add columns to cluster_df for LEFT, RIGHT, and COMMON
cluster_df['LEFT'] = cluster_df['gene'].apply(lambda x: x in LEFT_genes_set)
cluster_df['RIGHT'] = cluster_df['gene'].apply(lambda x: x in RIGHT_genes_set)
cluster_df['COMMON'] = cluster_df['gene'].apply(lambda x: x in COMMON_genes_set)

cluster_df
```

Example 142 (unknown):
```unknown
list_for_testing = cluster_df[(cluster_df['cluster_id']==clus_of_interest)&(cluster_df['LEFT']==True)].gene.tolist()
GO_plot(gene_list=list_for_testing, display_table=True, n_genes=n_genes)
```

Example 143 (unknown):
```unknown
MSigDB_Hallmark_2020
```

Example 144 (unknown):
```unknown
KEGG_2021_Human
```

Example 145 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 146 (unknown):
```unknown
Reactome_2022
```

Example 147 (unknown):
```unknown
list_for_testing = cluster_df[(cluster_df['cluster_id']==clus_of_interest)&(cluster_df['RIGHT']==True)].gene.tolist()
GO_plot(gene_list=list_for_testing, display_table=True, n_genes=n_genes)
```

Example 148 (unknown):
```unknown
MSigDB_Hallmark_2020
```

Example 149 (unknown):
```unknown
KEGG_2021_Human
```

Example 150 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 151 (unknown):
```unknown
Reactome_2022
```

Example 152 (unknown):
```unknown
list_for_testing = cluster_df[(cluster_df['cluster_id']==clus_of_interest)].gene.tolist()
GO_plot(gene_list=list_for_testing, display_table=True, n_genes=n_genes)
```

Example 153 (unknown):
```unknown
MSigDB_Hallmark_2020
```

Example 154 (unknown):
```unknown
KEGG_2021_Human
```

Example 155 (unknown):
```unknown
GO_Biological_Process_2023
```

Example 156 (unknown):
```unknown
Reactome_2022
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/Visium_mapping/c2l_pt2_mappings.html

**Contents:**
  - Loading packages¶
- Functions¶
- Read in metadata¶
- Data, parameters¶
- Loading visium data¶
- Filter out low quality spots¶
- Remove mitochondrial genes¶
- Add metadata and annotation¶
- Remove spots based on manual annotation¶
- Combine pcw13w-Atrial and pcw13w-Ventricle data to have whole heart in one sample¶
- Run per group: cell2location mapping¶

**Examples:**

Example 1 (python):
```python
import sys
import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
import matplotlib as mpl

import cell2location
import scvi

from matplotlib import rcParams
rcParams['pdf.fonttype'] = 42 # enables correct plotting of text
import seaborn as sns
```

Example 2 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Global seed set to 0
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  @numba.jit()
```

Example 3 (python):
```python
import gc
import torch
```

Example 4 (python):
```python
def read_and_qc(sample_name, path):
    r""" This function reads the data for one 10X spatial experiment into the anndata object.
    It also calculates QC metrics. Modify this function if required by your workflow.

    :param sample_name: Name of the sample
    :param path: path to data
    """

    adata = sc.read_visium(path + str(sample_name),
                           count_file='filtered_feature_bc_matrix.h5', load_images=True)
    adata.var_names_make_unique(join='-')
    
    adata.obs['sample'] = sample_name
    
    # rename genes to ENSEMBL
    adata.var['SYMBOL'] = adata.var_names
    adata.var_names = adata.var['gene_ids']
    adata.var_names.name = None

    # Calculate QC metrics
    sc.pp.calculate_qc_metrics(adata, inplace=True)
    adata.var['mt'] = [gene.startswith('MT-') for gene in adata.var['SYMBOL']]
    adata.obs['mt_frac'] = adata[:, adata.var['mt'].tolist()].X.sum(1).A.squeeze()/adata.obs['total_counts']

    # add sample name to obs names
    adata.obs["sample"] = [str(i) for i in adata.obs['sample']]
    adata.obs_names = adata.obs["sample"] \
                          + '_' + adata.obs_names
    adata.obs.index.name = 'spot_id'

    return adata

def select_slide(adata, s, s_col='sample'):
    r""" This function selects the data for one slide from the spatial anndata object.

    :param adata: Anndata object with multiple spatial experiments
    :param s: name of selected experiment
    :param s_col: column in adata.obs listing experiment name for each location
    """

    slide = adata[adata.obs[s_col].isin([s]), :]
    s_keys = list(slide.uns['spatial'].keys())
    s_spatial = np.array(s_keys)[[s in k for k in s_keys]][0]

    slide.uns['spatial'] = {s_spatial: slide.uns['spatial'][s_spatial]}

    return slide
```

Example 5 (python):
```python
meta=pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230208_for-cell2loc.csv',
                index_col=0)
meta
```

Example 6 (python):
```python
pd.crosstab(meta['donor'],meta['cell2loc-map_group'])
```

Example 7 (unknown):
```unknown
# select slides
# Healthy, OCT, and without DA
meta = meta[meta['cell2loc-map_group'].isin(['4PCW','5PCW','7PCW','13PCW','20PCW'])]

sampleIDs = list(meta['sangerID'])
len(sampleIDs)
```

Example 8 (unknown):
```unknown
run = "20Mar2024_fetal_OCT"
```

Example 9 (python):
```python
import os
results_folder = os.path.join(os.getcwd(),"analysis/Visium/cell2location_results")

ref_run_name = f'{results_folder}/reference_signatures/{run}'
run_name = f'{results_folder}/cell2location_map/{run}'
```

Example 10 (unknown):
```unknown
visium_path='/nfs/team205/heart/mapped/spaceranger110/'

celltype_key='fine_grain_modified'
```

Example 11 (python):
```python
# Read the data of the interested region into anndata objects
slides = []
for i in sampleIDs:
    slides.append(read_and_qc(i, path=visium_path))

# Combine anndata objects together
adata_vis = slides[0].concatenate(
    slides[1:],
    batch_key="sample",
    uns_merge="unique",
    batch_categories=sampleIDs,
    index_unique=None
)
```

Example 12 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1830: UserWarning: Variable names are not unique. To make them unique, call `.var_names_make_unique`.
  utils.warn_names_duplicates("var")
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/_core/anndata.py:1785: FutureWarning: X.dtype being converted to np.float32 from float64. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
  [AnnData(sparse.csr_matrix(a.shape), obs=a.obs) for a in all_adatas],
```

Example 13 (python):
```python
# PLOT QC FOR EACH SAMPLE
fig, axs = plt.subplots(len(slides), 4, figsize=(15, 4*len(slides)-4))
for i, s in enumerate(adata_vis.obs['sample'].unique()):
    #fig.suptitle('Covariates for filtering')

    slide = select_slide(adata_vis, s)
    sns.distplot(slide.obs['total_counts'],
                 kde=False, ax = axs[i, 0])
    axs[i, 0].set_xlim(0, adata_vis.obs['total_counts'].max())
    axs[i, 0].set_xlabel(f'total_counts | {s}')

    sns.distplot(slide.obs['total_counts']\
                 [slide.obs['total_counts']<2000],
                 kde=False, bins=40, ax = axs[i, 1])
    axs[i, 1].set_xlim(0, 2000)
    axs[i, 1].set_xlabel(f'total_counts | {s}')

    sns.distplot(slide.obs['n_genes_by_counts'],
                 kde=False, bins=60, ax = axs[i, 2])
    axs[i, 2].set_xlim(0, adata_vis.obs['n_genes_by_counts'].max())
    axs[i, 2].set_xlabel(f'n_genes_by_counts | {s}')

    sns.distplot(slide.obs['n_genes_by_counts']\
                 [slide.obs['n_genes_by_counts']<1000],
                 kde=False, bins=60, ax = axs[i, 3])
    axs[i, 3].set_xlim(0, 1000)
    axs[i, 3].set_xlabel(f'n_genes_by_counts | {s}')

plt.tight_layout()
```

Example 14 (python):
```python
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/anndata/compat/_overloaded_dict.py:106: ImplicitModificationWarning: Trying to modify attribute `._uns` of view, initializing view as actual.
  self.data[key] = value
/tmp/ipykernel_8715/2051608433.py:7: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts'],
/tmp/ipykernel_8715/2051608433.py:12: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['total_counts']\
/tmp/ipykernel_8715/2051608433.py:18: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts'],
/tmp/ipykernel_8715/2051608433.py:23: UserWarning: 

`distplot` is a deprecated function and will be removed in seaborn v0.14.0.

Please adapt your code to use either `displot` (a figure-level function with
similar flexibility) or `histplot` (an axes-level function for histograms).

For a guide to updating your code to use the new functions, please see
https://gist.github.com/mwaskom/de44147ed2974457ad6372750bbe5751

  sns.distplot(slide.obs['n_genes_by_counts']\
```

Example 15 (python):
```python
# filter out low quality spots
sc.pp.filter_cells(adata_vis, min_counts = 500)
sc.pp.filter_cells(adata_vis, min_genes = 300)

# Spot number per slide
adata_vis.obs['sample'].value_counts()
```

Example 16 (unknown):
```unknown
HCAHeartST13162330    2806
HCAHeartST11605165    2429
HCAHeartST13162331    2338
HCAHeartST13162329    1777
HCAHeartST10317190    1337
HCAHeartST11605166    1321
HCAHeartST10298116    1219
HCAHeartST10298120    1170
HCAHeartST10298117    1126
HCAHeartST11605168    1082
HCAHeartST11605167    1044
HCAHeartST10298115    1025
HCAHeartST10317189    1004
HCAHeartST10317184     897
HCAHeartST10298114     872
HCAHeartST10317186     855
HCAHeartST10317185     824
HCAHeartST13162332     505
Name: sample, dtype: int64
```

Example 17 (python):
```python
# find mitochondria-encoded (MT) genes
adata_vis.var['MT_gene'] = [gene.startswith('MT-') for gene in adata_vis.var['SYMBOL']]
print(sum(adata_vis.var['MT_gene']))
    
# remove MT genes for spatial mapping (keeping their counts in the object)
adata_vis.obsm['MT'] = adata_vis[:, adata_vis.var['MT_gene'].values].X.toarray()
adata_vis = adata_vis[:, ~adata_vis.var['MT_gene'].values]
print(sum(adata_vis.var['MT_gene']))
```

Example 18 (unknown):
```unknown
meta.head()
```

Example 19 (python):
```python
adata_vis.obs = adata_vis.obs.reset_index().merge(meta, how='left', left_on='sample', right_on='sangerID').set_index('spot_id')
adata_vis
```

Example 20 (unknown):
```unknown
AnnData object with n_obs × n_vars = 23631 × 33525
    obs: 'in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac', 'n_counts', 'n_genes', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status', 'cell2loc-map_group'
    var: 'gene_ids', 'feature_types', 'genome', 'SYMBOL', 'mt', 'n_cells_by_counts-HCAHeartST10298114', 'mean_counts-HCAHeartST10298114', 'log1p_mean_counts-HCAHeartST10298114', 'pct_dropout_by_counts-HCAHeartST10298114', 'total_counts-HCAHeartST10298114', 'log1p_total_counts-HCAHeartST10298114', 'n_cells_by_counts-HCAHeartST10298115', 'mean_counts-HCAHeartST10298115', 'log1p_mean_counts-HCAHeartST10298115', 'pct_dropout_by_counts-HCAHeartST10298115', 'total_counts-HCAHeartST10298115', 'log1p_total_counts-HCAHeartST10298115', 'n_cells_by_counts-HCAHeartST10298116', 'mean_counts-HCAHeartST10298116', 'log1p_mean_counts-HCAHeartST10298116', 'pct_dropout_by_counts-HCAHeartST10298116', 'total_counts-HCAHeartST10298116', 'log1p_total_counts-HCAHeartST10298116', 'n_cells_by_counts-HCAHeartST10298117', 'mean_counts-HCAHeartST10298117', 'log1p_mean_counts-HCAHeartST10298117', 'pct_dropout_by_counts-HCAHeartST10298117', 'total_counts-HCAHeartST10298117', 'log1p_total_counts-HCAHeartST10298117', 'n_cells_by_counts-HCAHeartST10298120', 'mean_counts-HCAHeartST10298120', 'log1p_mean_counts-HCAHeartST10298120', 'pct_dropout_by_counts-HCAHeartST10298120', 'total_counts-HCAHeartST10298120', 'log1p_total_counts-HCAHeartST10298120', 'n_cells_by_counts-HCAHeartST10317184', 'mean_counts-HCAHeartST10317184', 'log1p_mean_counts-HCAHeartST10317184', 'pct_dropout_by_counts-HCAHeartST10317184', 'total_counts-HCAHeartST10317184', 'log1p_total_counts-HCAHeartST10317184', 'n_cells_by_counts-HCAHeartST10317185', 'mean_counts-HCAHeartST10317185', 'log1p_mean_counts-HCAHeartST10317185', 'pct_dropout_by_counts-HCAHeartST10317185', 'total_counts-HCAHeartST10317185', 'log1p_total_counts-HCAHeartST10317185', 'n_cells_by_counts-HCAHeartST10317186', 'mean_counts-HCAHeartST10317186', 'log1p_mean_counts-HCAHeartST10317186', 'pct_dropout_by_counts-HCAHeartST10317186', 'total_counts-HCAHeartST10317186', 'log1p_total_counts-HCAHeartST10317186', 'n_cells_by_counts-HCAHeartST10317189', 'mean_counts-HCAHeartST10317189', 'log1p_mean_counts-HCAHeartST10317189', 'pct_dropout_by_counts-HCAHeartST10317189', 'total_counts-HCAHeartST10317189', 'log1p_total_counts-HCAHeartST10317189', 'n_cells_by_counts-HCAHeartST10317190', 'mean_counts-HCAHeartST10317190', 'log1p_mean_counts-HCAHeartST10317190', 'pct_dropout_by_counts-HCAHeartST10317190', 'total_counts-HCAHeartST10317190', 'log1p_total_counts-HCAHeartST10317190', 'n_cells_by_counts-HCAHeartST11605165', 'mean_counts-HCAHeartST11605165', 'log1p_mean_counts-HCAHeartST11605165', 'pct_dropout_by_counts-HCAHeartST11605165', 'total_counts-HCAHeartST11605165', 'log1p_total_counts-HCAHeartST11605165', 'n_cells_by_counts-HCAHeartST11605166', 'mean_counts-HCAHeartST11605166', 'log1p_mean_counts-HCAHeartST11605166', 'pct_dropout_by_counts-HCAHeartST11605166', 'total_counts-HCAHeartST11605166', 'log1p_total_counts-HCAHeartST11605166', 'n_cells_by_counts-HCAHeartST11605167', 'mean_counts-HCAHeartST11605167', 'log1p_mean_counts-HCAHeartST11605167', 'pct_dropout_by_counts-HCAHeartST11605167', 'total_counts-HCAHeartST11605167', 'log1p_total_counts-HCAHeartST11605167', 'n_cells_by_counts-HCAHeartST11605168', 'mean_counts-HCAHeartST11605168', 'log1p_mean_counts-HCAHeartST11605168', 'pct_dropout_by_counts-HCAHeartST11605168', 'total_counts-HCAHeartST11605168', 'log1p_total_counts-HCAHeartST11605168', 'n_cells_by_counts-HCAHeartST13162329', 'mean_counts-HCAHeartST13162329', 'log1p_mean_counts-HCAHeartST13162329', 'pct_dropout_by_counts-HCAHeartST13162329', 'total_counts-HCAHeartST13162329', 'log1p_total_counts-HCAHeartST13162329', 'n_cells_by_counts-HCAHeartST13162330', 'mean_counts-HCAHeartST13162330', 'log1p_mean_counts-HCAHeartST13162330', 'pct_dropout_by_counts-HCAHeartST13162330', 'total_counts-HCAHeartST13162330', 'log1p_total_counts-HCAHeartST13162330', 'n_cells_by_counts-HCAHeartST13162331', 'mean_counts-HCAHeartST13162331', 'log1p_mean_counts-HCAHeartST13162331', 'pct_dropout_by_counts-HCAHeartST13162331', 'total_counts-HCAHeartST13162331', 'log1p_total_counts-HCAHeartST13162331', 'n_cells_by_counts-HCAHeartST13162332', 'mean_counts-HCAHeartST13162332', 'log1p_mean_counts-HCAHeartST13162332', 'pct_dropout_by_counts-HCAHeartST13162332', 'total_counts-HCAHeartST13162332', 'log1p_total_counts-HCAHeartST13162332', 'MT_gene'
    uns: 'spatial'
    obsm: 'spatial', 'MT'
```

Example 21 (python):
```python
adata_vis.obs.head()
```

Example 22 (python):
```python
# annotation
path_to_annot='/nfs/team205/heart/visium_annotations/foetal'
category_name='JC_edit'

# read in annotation csv file and concatenate
for i, ID in enumerate(sampleIDs):
    if f'{ID}_JC.csv' in os.listdir(path_to_annot):
    
        print(ID)
        annot_df = pd.read_csv(f'{path_to_annot}/{ID}_JC.csv')
        annot_df['spot_id']=ID+'_'+annot_df['Barcode']

        if i==0:
            annot_df_all=annot_df.copy()
        else:
            annot_df_all=pd.concat([annot_df_all,annot_df])
    else:
        print(f'{ID} is not annotated')

# add annotations to adata_vis
adata_vis.obs=pd.concat([adata_vis.obs,
                     annot_df_all.set_index('spot_id')[[category_name]].reindex(adata_vis.obs.index)],
                     axis=1)
# replace nan to 'none'
adata_vis.obs[category_name].fillna('none', inplace=True)

# minor modification
annotation_edit_dict = {
     'AV Ring':'AV ring',
    'Aorta':'Great vessels',
    'Coronary Artery':'Coronary artery',
    'Mitral Valve':'AV valves',
    'Mitral valve':'AV valves',
    'Myocardium - Ventricular - Trabecular':'Myocardium - Ventricular - Trabeculated',
    'Myocardim - Ventricular - Trabeculated':'Myocardium - Ventricular - Trabeculated',
    'Myocardim - Ventricular - Compact':'Myocardium - Ventricular - Compact',
    'unknown':'Unannotated',
    'Unknown':'Unannotated',
}
adata_vis.obs.replace({category_name:annotation_edit_dict}, inplace=True)
set(adata_vis.obs[category_name])
```

Example 23 (unknown):
```unknown
HCAHeartST10298114
HCAHeartST10298115
HCAHeartST10298116
HCAHeartST10298117
HCAHeartST10298120
HCAHeartST10317184
HCAHeartST10317185
HCAHeartST10317186
HCAHeartST10317189
HCAHeartST10317190
HCAHeartST11605165
HCAHeartST11605166
HCAHeartST11605167
HCAHeartST11605168
HCAHeartST13162329
HCAHeartST13162330
HCAHeartST13162331
HCAHeartST13162332
```

Example 24 (unknown):
```unknown
{'AV ring',
 'AV valves',
 'Aortic valve',
 'Cavity',
 'Connective tissue',
 'Coronary artery',
 'Epicardium',
 'Great vessels',
 'Myocardium - Atrial',
 'Myocardium - Papillary Muscle',
 'Myocardium - Ventricular - Compact',
 'Myocardium - Ventricular - Trabeculated',
 'Out_of_tissue',
 'Unannotated',
 'Valve apparatus',
 'none'}
```

Example 25 (python):
```python
# remove 'none' and 'no_tissue'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['none','Out_of_tissue'])==False]

# remove 'Cavity'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['Cavity'])==False]

# remove 'Unannotated'
adata_vis = adata_vis[adata_vis.obs[category_name].isin(['Unannotated'])==False]
```

Example 26 (python):
```python
# order annotations
annotations_order = ['Myocardium - Atrial',
                     'Myocardium - Ventricular - Compact',
                     'Myocardium - Ventricular - Trabeculated',
                    'Myocardium - Papillary Muscle',
                    'Epicardium',
                     'AV valves',
                     'Aortic valve',
                    'Valve apparatus',
                   'AV ring',
                    'Connective tissue',
                     'Great vessels',
                     'Coronary artery',
                     # 'Nerve',
                     ]
adata_vis.obs[category_name] = adata_vis.obs[category_name].astype('category')
adata_vis.obs[category_name].cat.reorder_categories(annotations_order,inplace=True)
```

Example 27 (python):
```python
/tmp/ipykernel_8715/3572501826.py:16: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
  adata_vis.obs[category_name] = adata_vis.obs[category_name].astype('category')
/tmp/ipykernel_8715/3572501826.py:17: FutureWarning: The `inplace` parameter in pandas.Categorical.reorder_categories is deprecated and will be removed in a future version. Reordering categories will always return a new Categorical object.
  adata_vis.obs[category_name].cat.reorder_categories(annotations_order,inplace=True)
```

Example 28 (python):
```python
pd.crosstab(adata_vis.obs['cell2loc-map_group'],adata_vis.obs[category_name])
```

Example 29 (python):
```python
# check the slides for 'PCW-13w'
mask = adata_vis.obs['age']=='PCW-13w'
adata_vis[mask].obs['sample'].value_counts()
# 'HCAHeartST11605166': Atrium
# 'HCAHeartST11605165': Ventricle
```

Example 30 (unknown):
```unknown
HCAHeartST11605165    2428
HCAHeartST11605166    1120
Name: sample, dtype: int64
```

Example 31 (python):
```python
# combine
adata_vis.obs['sample_for_cell2loc']=adata_vis.obs['sample'].astype('str')
adata_vis.obs.replace({'sample_for_cell2loc':{
    'HCAHeartST11605165':'HCAHeartST11605165_HCAHeartST11605166',
    'HCAHeartST11605166':'HCAHeartST11605165_HCAHeartST11605166'
}},inplace=True)
adata_vis.obs['sample_for_cell2loc'].value_counts()
```

Example 32 (unknown):
```unknown
HCAHeartST11605165_HCAHeartST11605166    3548
HCAHeartST13162330                       2806
HCAHeartST13162331                       2338
HCAHeartST13162329                       1623
HCAHeartST10298116                       1187
HCAHeartST10298120                       1108
HCAHeartST10298117                       1085
HCAHeartST10298115                        991
HCAHeartST10317189                        944
HCAHeartST10317184                        836
HCAHeartST10298114                        812
HCAHeartST10317185                        704
HCAHeartST10317186                        693
HCAHeartST11605167                        692
HCAHeartST11605168                        639
HCAHeartST10317190                        609
HCAHeartST13162332                        413
Name: sample_for_cell2loc, dtype: int64
```

Example 33 (python):
```python
adata_vis.obs.columns
```

Example 34 (unknown):
```unknown
Index(['in_tissue', 'array_row', 'array_col', 'sample', 'n_genes_by_counts',
       'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts',
       'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes',
       'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'mt_frac',
       'n_counts', 'n_genes', 'sangerID', 'combinedID', 'donor', 'donor_type',
       'region', 'region_finest', 'age', 'gender', 'facility',
       'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'status',
       'cell2loc-map_group', 'JC_edit', 'sample_for_cell2loc'],
      dtype='object')
```

Example 35 (unknown):
```unknown
set(meta['cell2loc-map_group'])
```

Example 36 (unknown):
```unknown
{'13PCW', '20PCW', '4PCW', '5PCW', '7PCW'}
```

Example 37 (python):
```python
# convert adata_vis.var_names to gene_id
adata_vis.var.set_index('gene_ids',inplace=True)
adata_vis.var.head()
```

Example 38 (python):
```python
%%time
import cell2location
# read in reference signatures
inf_aver = pd.read_csv(f"{ref_run_name}/all_data/inf_aver.csv", index_col=0)

# find shared genes and subset both anndata and reference signatures
intersect = np.intersect1d(adata_vis.var_names, inf_aver.index)
adata_vis = adata_vis[:, intersect].copy()
inf_aver = inf_aver.loc[intersect, :].copy()
print(inf_aver.shape)

for group in set(meta['cell2loc-map_group']):
    print(f'####### {group} #######')
    
    # subset adata
    adata_vis_sub=adata_vis[adata_vis.obs['cell2loc-map_group']==group].copy()
    samples = list(set(adata_vis_sub.obs['sample_for_cell2loc']))
    samples.sort()
    print(f'samples in this run: {samples}')
    adata_vis_sub=adata_vis_sub.copy()
    # prepare anndata for cell2location model
    cell2location.models.Cell2location.setup_anndata(adata=adata_vis_sub, batch_key="sample_for_cell2loc")
    #scvi.data.setup_anndata(adata=adata_vis_sub, batch_key="sample_for_cell2loc")
    #scvi.data.view_anndata_setup(adata_vis_sub)

    # create and train the model
    mod = cell2location.models.Cell2location(
        adata_vis_sub, cell_state_df=inf_aver, 
        # the expected average cell abundance: tissue-dependent 
        # hyper-prior which can be estimated from paired histology:
        N_cells_per_location=20,
        # hyperparameter controlling normalisation of
        # within-experiment variation in RNA detection (using default here):
        detection_alpha=20
    ) 

    mod.train(max_epochs=20000, 
              # train using full data (batch_size=None)
              batch_size=None, 
              # use all data points in training because 
              # we need to estimate cell abundance at all locations
              train_size=1,
              use_gpu=True)
    mod.view_anndata_setup()
    # plot ELBO loss history during training, removing first 100 epochs from the plot
    mod.plot_history(1000)
    plt.legend(labels=['full data training']);

    # In this section, we export the estimated cell abundance (summary of the posterior distribution).
    adata_vis_sub = mod.export_posterior(
        adata_vis_sub, sample_kwargs={'num_samples': 1000, 'batch_size': mod.adata.n_obs, 'use_gpu': True}
    )

    # Save model
    mod.save(f"{run_name}/{group}", overwrite=True)

    # Save anndata object with results
    adata_vis_sub.write(f"{run_name}/{group}/sp.h5ad")

    # Examine reconstruction accuracy to assess if there are any issues with mapping
    # the plot should be roughly diagonal, strong deviations will signal problems
    mod.plot_QC()
    
    del mod,adata_vis_sub
    gc.collect()
    torch.cuda.empty_cache()
```

Example 39 (unknown):
```unknown
(17072, 64)
####### 5PCW #######
samples in this run: ['HCAHeartST11605167', 'HCAHeartST11605168']
```

Example 40 (unknown):
```unknown
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/configuration_validator.py:106: UserWarning: You passed in a `val_dataloader` but have no `validation_step`. Skipping val loop.
  rank_zero_warn("You passed in a `val_dataloader` but have no `validation_step`. Skipping val loop.")
You are using a CUDA device ('NVIDIA A100-SXM4-80GB') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omniGPU_env/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py:1609: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=10). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
  rank_zero_warn(
```

Example 41 (unknown):
```unknown
Epoch 1511/20000:   8%|███████████▍                                                                                                                                           | 1511/20000 [00:58<11:30, 26.77it/s, v_num=1, elbo_train=1.84e+7]
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/TransmuralAxis/tm-axis_cell2loc_t21.html

**Contents:**
- Import modules¶
- Functions¶
- Read in adata¶
- Add axis¶
- Plot transmural axis¶
- Plot cells along the axis and time¶

**Examples:**

Example 1 (python):
```python
import os
import numpy as np
import scanpy as sc
import anndata
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 
import seaborn as sns

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter("ignore", UserWarning)
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
anndata             0.9.2
matplotlib          3.7.2
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                         10.0.0
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.11.17
cffi                        1.15.1
charset_normalizer          3.2.0
colorama                    0.4.6
comm                        0.1.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.8
decorator                   5.1.1
defusedxml                  0.7.1
executing                   1.2.0
fastjsonschema              NA
fqdn                        NA
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_metadata          NA
importlib_resources         NA
ipykernel                   6.25.1
isoduration                 NA
jedi                        0.19.0
jinja2                      3.1.2
joblib                      1.3.2
json5                       NA
jsonpointer                 2.0
jsonschema                  4.19.0
jsonschema_specifications   NA
jupyter_events              0.7.0
jupyter_server              2.7.1
jupyterlab_server           2.24.0
kiwisolver                  1.4.5
leidenalg                   0.10.1
llvmlite                    0.40.1
markupsafe                  2.1.3
mpl_toolkits                NA
natsort                     8.4.0
nbformat                    5.9.2
numba                       0.57.1
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.16.1
pyparsing                   3.0.9
pythonjsonlogger            NA
pytz                        2023.3
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
ruamel                      NA
scipy                       1.10.1
send2trash                  NA
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
sphinxcontrib               NA
stack_data                  0.6.2
statsmodels                 0.14.0
texttable                   1.6.7
threadpoolctl               3.2.0
tornado                     6.3.3
traitlets                   5.9.0
typing_extensions           NA
uri_template                NA
urllib3                     2.0.4
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.2
yaml                        6.0.1
zipp                        NA
zmq                         25.1.1
zstandard                   0.19.0
```

Example 5 (unknown):
```unknown
-----
IPython             8.12.0
jupyter_client      8.3.0
jupyter_core        4.12.0
jupyterlab          4.0.5
notebook            7.0.2
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-4.15.0-220-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-02-15 10:02
```

Example 6 (python):
```python
import importlib.util
import sys

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/tissueaxis.py")
tissueaxis = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = tissueaxis
spec.loader.exec_module(tissueaxis)
```

Example 7 (python):
```python
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 8 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 9 (python):
```python
# This function selects the data for multiple slides from the spatial anndata object.
# Modified `select_slide` in cell2loc for multiple slides

# @param adata: anndata object with multiple spatial experiments
# @param s: list of selected slides
# @param s_col: column in adata.obs listing experiment name for each location

def select_multislide(adata, s, s_col='sample'):

    slides = adata[adata.obs[s_col].isin(s), :]
    s_keys = list(slides.uns['spatial'].keys())
    
    uns_dict = {}
    for ss in s:
        s_spatial=np.array(s_keys)[[ss in k for k in s_keys]][0]
        uns_dict[s_spatial]=slides.uns['spatial'][s_spatial]
    
    slides.uns['spatial'] = uns_dict

    return slides
```

Example 10 (python):
```python
adata_oct_healthy = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT_foetal_raw.h5ad')
print(adata_oct_healthy.shape)
adata_oct_t21 = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT-T21_foetal_raw.h5ad')
print(adata_oct_t21.shape)
```

Example 11 (unknown):
```unknown
(14261, 33538)
(11238, 33538)
```

Example 12 (python):
```python
# concatenate oct adata
adata_oct_healthy.obs['status'] = 'Euploid'
adata_oct_t21.obs['status'] = 'Trisomy21'

adata_oct = adata_oct_healthy.concatenate(
    [adata_oct_t21],
    batch_key=None,
    index_unique=None,
    join="inner"
)

adata_oct.obs['status'].value_counts()
```

Example 13 (unknown):
```unknown
Euploid      14261
Trisomy21    11238
Name: status, dtype: int64
```

Example 14 (python):
```python
# log-normalise
sc.pp.normalize_total(adata_oct, target_sum=1e4)
sc.pp.log1p(adata_oct)

# reset var_names
adata_oct.var.set_index('SYMBOL',inplace=True)
```

Example 15 (python):
```python
# get all celltypes
all_celltypes = adata_oct.obsm['q05_cell_abundance_w_sf'].columns
all_celltypes = [x.replace('q05cell_abundance_w_sf_','') for x in all_celltypes]
print(len(all_celltypes))
print(all_celltypes[:5])
```

Example 16 (unknown):
```unknown
54
['AtrialCardiomyocytesLeft', 'AtrialCardiomyocytesRight', 'VentricularCardiomyocytesLeftCompact', 'VentricularCardiomyocytesRightCompact', 'VentricularCardiomyocytesLeftTrabeculated']
```

Example 17 (python):
```python
# subset cell types in ffpe
adata_oct.obsm['q05_cell_abundance_w_sf'] = adata_oct.obsm['q05_cell_abundance_w_sf'][[f'q05cell_abundance_w_sf_{x}' for x in all_celltypes]]

# modify cell type names
celltype_rename = utils.read_pkl('/nfs/team205/heart/anndata_objects/Foetal/fine_grain_rename_dictionary.pkl')
adata_oct.obsm['q05_cell_abundance_w_sf'].columns = [x.replace(x.split('q05cell_abundance_w_sf_')[1],celltype_rename[x.split('q05cell_abundance_w_sf_')[1]]) \
                                                      for x in adata_oct.obsm['q05_cell_abundance_w_sf'].columns]
```

Example 18 (python):
```python
# add
oct_samples = list(set(adata_oct.obs['sample']))

adata_oct.obs['transmural_axis'] = np.nan

axis_files = os.listdir('/nfs/team205/heart/visium_axis/foetal/transmural_LV')
for file in axis_files:
    # read in
    axis = pd.read_csv(f'/nfs/team205/heart/visium_axis/foetal/transmural_LV/{file}',index_col=0)
    # add to anndata
    if file.split('.csv')[0] in oct_samples:
        adata_oct.obs.loc[axis.index,'transmural_axis'] = axis.copy()
        if file.split('.csv')[0]=='HCAHeartST11605165':
            adata_oct_healthy.obs.loc[axis.index,'transmural_axis'] = axis.copy()
        elif file.split('.csv')[0]=='HCAHeartST13162336':
            adata_oct_t21.obs.loc[axis.index,'transmural_axis'] = axis.copy()

# select spots with axis
adata_oct = adata_oct[adata_oct.obs['transmural_axis'].isna()==False]
```

Example 19 (python):
```python
pd.crosstab(adata_oct.obs['sample'],adata_oct.obs['cell2loc-map_group'])
```

Example 20 (python):
```python
slide_id = 'HCAHeartST11605165'
slide = select_multislide(adata_oct_healthy, [slide_id])
sc.pl.spatial(slide,color=['transmural_axis'],cmap='gist_rainbow',show=False)
plt.savefig(f"{figdir}/visium_tm-axis_euploid_{slide_id}.pdf", bbox_inches="tight",dpi=300)
```

Example 21 (python):
```python
slide_id = 'HCAHeartST13162336'
slide = select_multislide(adata_oct_t21, [slide_id])
sc.pl.spatial(slide,color=['transmural_axis'],cmap='gist_rainbow',show=False)
plt.savefig(f"{figdir}/visium_tm-axis_t21_{slide_id}.pdf", bbox_inches="tight",dpi=300)
```

Example 22 (python):
```python
features_to_plot = ['VentricularCardiomyocytesLeftCompact',
                    'CoronaryCapillaryEndothelialCells','CoronaryPericytes'
                   ]

for feature in features_to_plot:
    tissueaxis.plot_axis_split(adata_oct,
                               feature_to_plot=feature,
                               feature_type='celltype',
                               min_max_scale=False,
                               splitby='cell2loc-map_group',
                               groups_order=['13PCW','T21_13PCW'],
                               figsize=(5, 3),
                               cbar_pos=(0, 0.7, 0.05, 0.18),
                               save=f'{figdir}/tm-axis_cell2loc_t21_{feature}.pdf'
                              )
```

Example 23 (unknown):
```unknown
?tissueaxis.plot_axis_split
```

Example 24 (python):
```python
Signature:
tissueaxis.plot_axis_split(
    adata,
    feature_to_plot,
    splitby,
    groups_order=None,
    axis_col='transmural_axis',
    n_bin=15,
    feature_type='gene',
    min_max_scale=True,
    row_cluster=False,
    figsize=(3, 8),
    cbar_pos=(0, 0.5, 0.05, 0.18),
    cmap='viridis',
    xticklabels=True,
    yticklabels=True,
    title=None,
    save=None,
    save_dpi=300,
    **kwargs,
)
Docstring: <no docstring>
File:      /nfs/team205/kk18/function/python/tissueaxis.py
Type:      function
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/CellOracle/CellOracle_no3_summarise-output.html

**Contents:**
- Import libraries¶
- Parameters¶
- Read in Oracle object¶
- Read in celloracle in-silico perturbation output¶
- Annotate grids¶
- Plot heatmap of inner-product score¶

ref: https://morris-lab.github.io/CellOracle.documentation/notebooks/04_Network_analysis/Network_analysis_with_Paul_etal_2015_data.html

First we need to align cells and grids

Next, lets give ID for each grid and make sure we are assigning the calculated inner product score correctly

Next step is to annotate each grid

5664 rows × 3 columns

6777 rows × 6 columns

**Examples:**

Example 1 (python):
```python
import os
import sys

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scanpy as sc
import seaborn as sns

import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42
```

Example 2 (python):
```python
import celloracle as co
co.__version__
```

Example 3 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/regulon_plot.py")
regulon_plot = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = regulon_plot
spec.loader.exec_module(regulon_plot)
```

Example 4 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 5 (python):
```python
%matplotlib inline
```

Example 6 (unknown):
```unknown
!ls /nfs/team205/heart/anndata_objects/Foetal/celloracle
```

Example 7 (unknown):
```unknown
archive_AtrialCM-Left_in-silico_perturbation_May2025.pkl
archive_AtrialCM-Right_in-silico_perturbation_May2025.pkl
AtrialCM_in-silico_perturbation_May2025.pkl
AtrialCM-Right_in-silico_perturbation_May2025.pkl
figures_pm
foetal_AtrialCM.celloracle.oracle
foetal_AtrialCM_links.celloracle.links
```

Example 8 (unknown):
```unknown
# Parameters
# working_dir = "/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/CO_in-silico-perturbation_papermill"
path_to_oracle_object = "/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM.celloracle.oracle"
path_to_oracle_links = "/nfs/team205/heart/anndata_objects/Foetal/celloracle/foetal_AtrialCM_links.celloracle.links"
n_grid = 40
min_mass = 0.011
cell_group_name = "AtrialCM"
path_to_res_dict = f"/nfs/team205/heart/anndata_objects/Foetal/celloracle/{cell_group_name}_in-silico_perturbation_May2025.pkl"
```

Example 9 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk/Aug2024'
table_dir = '/nfs/team205/heart/fetal_ms_figures/tables/Aug2024'
```

Example 10 (unknown):
```unknown
# Load
oracle = co.load_hdf5(path_to_oracle_object)
```

Example 11 (python):
```python
%matplotlib inline
# Check clustering data
celltype_col = 'fine_grain_time-binned'
sc.pl.umap(oracle.adata, color=celltype_col)
```

Example 12 (python):
```python
dev_dict = utils.read_pkl(path_to_res_dict)
print(len(dev_dict.keys()))
dev_dict.keys()
```

Example 13 (unknown):
```unknown
dict_keys(['ETS2', 'MAF', 'IKZF2', 'ETV6', 'BHLHE41', 'GTF2IRD1', 'GATA5', 'PITX2', 'SIX5'])
```

Example 14 (unknown):
```unknown
?dev.visualize_development_module_layout_0
```

Example 15 (unknown):
```unknown
Object `dev.visualize_development_module_layout_0` not found.
```

Example 16 (python):
```python
%matplotlib inline
# Hide all INFO (and DEBUG) messages from Matplotlib
import logging
logging.getLogger("matplotlib").setLevel(logging.WARNING)

vm = 0.2
scale_simulation = 5
scale_dev = 40

for tf in dev_dict.keys():
    dev = dev_dict[tf]
    print(f'##### {tf} #####')
    dev.visualize_development_module_layout_0(s=5,
                                              scale_for_simulation=scale_simulation,
                                              s_grid=50,
                                              scale_for_pseudotime=scale_dev,
                                              vm=vm)
    plt.show()
```

Example 17 (unknown):
```unknown
##### ETS2 #####
```

Example 18 (unknown):
```unknown
##### MAF #####
```

Example 19 (unknown):
```unknown
##### IKZF2 #####
```

Example 20 (unknown):
```unknown
##### ETV6 #####
```

Example 21 (unknown):
```unknown
##### BHLHE41 #####
```

Example 22 (unknown):
```unknown
##### GTF2IRD1 #####
```

Example 23 (unknown):
```unknown
##### GATA5 #####
```

Example 24 (unknown):
```unknown
##### PITX2 #####
```

Example 25 (unknown):
```unknown
##### SIX5 #####
```

Example 26 (unknown):
```unknown
# pick one tf for assessing the alignment is correct
tf = 'PITX2'
dev = dev_dict[tf]
```

Example 27 (python):
```python
## let's make sure the obtained coordinates are making sense
# cell embedding
cell_embedding_coords = oracle.adata.obsm['X_umap'].copy()
sns.scatterplot(x=cell_embedding_coords[:,0],
                y=cell_embedding_coords[:,1],
                s=3
               )
plt.title('cell embedding')
plt.show()

# grid embedding
grid_embedding_coords = dev.gridpoints_coordinates[~dev.mass_filter_reference]
sns.scatterplot(x=grid_embedding_coords[:,0],
                y=grid_embedding_coords[:,1],
                s=25
               )
plt.title('grid embedding')
plt.show()
```

Example 28 (python):
```python
# groundtruth plot, same as above
print(f'##### {tf} #####')
dev.plot_inner_product_on_grid(vm=vm)
plt.show()
```

Example 29 (unknown):
```unknown
##### PITX2 #####
```

Example 30 (python):
```python
# align score with coordinates
inner_product_df = dev.inner_product_df.copy()
inner_product_df[['x','y']] = dev.gridpoints_coordinates[~dev.mass_filter_reference]

from matplotlib.colors import TwoSlopeNorm
x = inner_product_df['x']
y = inner_product_df['y']
c = inner_product_df['score']

# set up the norm so that 0 is centered
norm = TwoSlopeNorm(vmin=c.min(),
                    vcenter=0,
                    vmax=c.max())

sns.scatterplot(x=x,y=y,c=c,cmap='PiYG',hue_norm=norm,s=25)
plt.show()
```

Example 31 (python):
```python
celltype_df = oracle.adata.obs[[celltype_col]].copy()
celltype_df[['x','y']] = oracle.adata.obsm['X_umap'].copy()
celltype_df
```

Example 32 (python):
```python
# annotate grids
import pandas as pd
import numpy as np
from sklearn.neighbors import NearestNeighbors

def annotate_grid(cells_df, grid_df, k, celltype_col):
    """
    Annotate each grid point with the nearest cell_type label from cells_df.
    
    Parameters:
    - cells_df: DataFrame with columns ['x', 'y', celltype_col]
    - grid_df:  DataFrame with columns ['x', 'y']
    - k:        Number of nearest neighbors to consider (majority vote if k > 1)
    
    Returns:
    - grid_df with an added 'cell_type' column.
    """
    coords = cells_df[['x', 'y']].values
    nbrs = NearestNeighbors(n_neighbors=k).fit(coords)
    distances, indices = nbrs.kneighbors(grid_df[['x', 'y']].values)
    
    if k == 1:
        grid_df[celltype_col] = cells_df[celltype_col].values[indices.flatten()]
    else:
        # Majority vote over k neighbors
        labels = cells_df[celltype_col].values[indices]  # shape (n_grid, k)
        majority = []
        for row in labels:
            uniq, counts = np.unique(row, return_counts=True)
            majority.append(uniq[np.argmax(counts)])
        grid_df[celltype_col] = majority
    
    return grid_df
```

Example 33 (python):
```python
# to decide k
celltype_df.shape[0]/inner_product_df.shape[0]
```

Example 34 (unknown):
```unknown
7.52191235059761
```

Example 35 (unknown):
```unknown
celltype_labels = annotate_grid(cells_df=celltype_df, 
                                  grid_df=inner_product_df, 
                                  k=7,
                                  celltype_col=celltype_col)[celltype_col]
celltype_labels
```

Example 36 (unknown):
```unknown
0       AtrialCardiomyocytesLeft_t3
1       AtrialCardiomyocytesLeft_t3
2       AtrialCardiomyocytesLeft_t3
3       AtrialCardiomyocytesLeft_t3
4       AtrialCardiomyocytesLeft_t3
                   ...             
748     AtrialCardiomyocytesLeft_t1
749     AtrialCardiomyocytesLeft_t1
750     AtrialCardiomyocytesLeft_t1
751    AtrialCardiomyocytesRight_t2
752    AtrialCardiomyocytesRight_t2
Name: fine_grain_time-binned, Length: 753, dtype: object
```

Example 37 (python):
```python
x = inner_product_df['x']
y = inner_product_df['y']
c = celltype_labels
sns.scatterplot(x=x,y=y,hue=c,s=25)
plt.legend(bbox_to_anchor=(1, 0.5))
plt.show()
```

Example 38 (python):
```python
# update cell type name
df = pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/finegrain_name_mapping.csv')
celltype_mapping = df.set_index('Full_name')['Short_name_finalised'].to_dict()
celltype_labels_mod = [f'{celltype_mapping[x.split("_")[0]]}_{x.split("_")[1]}' for x in celltype_labels]
celltype_labels_mod[:5]
```

Example 39 (unknown):
```unknown
['aCML_t3', 'aCML_t3', 'aCML_t3', 'aCML_t3', 'aCML_t3']
```

Example 40 (python):
```python
# read in post-g2g-selection csv file and get selected TFs
# log2FC<0 means TFs specific to Right-aCM
# log2FC>0 means TFs specific to Left-aCM
g2g_sel = pd.read_csv('/nfs/team205/kk18/notebooks/Foetal/AtrialCM_celloracle/AtrialCM_post-g2g_selected-recgulons.csv')
display(g2g_sel)

# TFs, only activator
list_of_TFs = [x.split('_')[0] for x in g2g_sel['Gene'] if '_+' in x]
print(len(list_of_TFs))
```

Example 41 (python):
```python
for i,tf in enumerate(dev_dict.keys()):
    dev = dev_dict[tf]
    inner_product_df = dev.inner_product_df.copy()
    inner_product_df['TF'] = tf
    inner_product_df[celltype_col] = celltype_labels_mod
    if i==0:
        inner_product_df_all = inner_product_df.copy()
    else:
        inner_product_df_all = pd.concat([inner_product_df_all,inner_product_df])
inner_product_df_all
```

Example 42 (python):
```python
aggr_df = inner_product_df_all[['score','TF',celltype_col]].groupby(['TF',celltype_col]).sum() # "sum", so this is reflecting the number of grids (cells)
aggr_df = aggr_df.reset_index()
df = aggr_df.pivot(index='TF',columns=celltype_col,values='score')
# reorder based on g2g-log2FC values
df = df.loc[list_of_TFs.rev]

plt.rcParams['figure.figsize'] = 3,3
sns.heatmap(df, center=0, cmap='PiYG')
plt.savefig(f"{figdir}/AtrialCM_CellOracle_heatmap.pdf", bbox_inches="tight",dpi=300)
plt.show()
```

Example 43 (unknown):
```unknown
2025-05-13 17:05:12,591 - INFO - maxp pruned
2025-05-13 17:05:12,601 - INFO - cmap pruned
2025-05-13 17:05:12,603 - INFO - kern dropped
2025-05-13 17:05:12,604 - INFO - post pruned
2025-05-13 17:05:12,604 - INFO - FFTM dropped
2025-05-13 17:05:12,606 - INFO - GPOS pruned
2025-05-13 17:05:12,614 - INFO - GSUB pruned
2025-05-13 17:05:12,622 - INFO - glyf pruned
2025-05-13 17:05:12,625 - INFO - Added gid0 to subset
2025-05-13 17:05:12,627 - INFO - Added first four glyphs to subset
2025-05-13 17:05:12,627 - INFO - Closing glyph list over 'MATH': 44 glyphs before
2025-05-13 17:05:12,628 - INFO - Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'K', 'L', 'M', 'P', 'R', 'S', 'T', 'V', 'X', 'Z', 'a', 'b', 'd', 'e', 'f', 'five', 'four', 'g', 'hyphen', 'i', 'm', 'minus', 'n', 'nonmarkingreturn', 'one', 'r', 'six', 'space', 't', 'three', 'two', 'underscore', 'zero']
2025-05-13 17:05:12,629 - INFO - Glyph IDs:   [0, 1, 2, 3, 16, 19, 20, 21, 22, 23, 24, 25, 36, 37, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 51, 53, 54, 55, 57, 59, 61, 66, 68, 69, 71, 72, 73, 74, 76, 80, 81, 85, 87, 3228]
2025-05-13 17:05:12,633 - INFO - Closed glyph list over 'MATH': 44 glyphs after
2025-05-13 17:05:12,634 - INFO - Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'K', 'L', 'M', 'P', 'R', 'S', 'T', 'V', 'X', 'Z', 'a', 'b', 'd', 'e', 'f', 'five', 'four', 'g', 'hyphen', 'i', 'm', 'minus', 'n', 'nonmarkingreturn', 'one', 'r', 'six', 'space', 't', 'three', 'two', 'underscore', 'zero']
2025-05-13 17:05:12,634 - INFO - Glyph IDs:   [0, 1, 2, 3, 16, 19, 20, 21, 22, 23, 24, 25, 36, 37, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 51, 53, 54, 55, 57, 59, 61, 66, 68, 69, 71, 72, 73, 74, 76, 80, 81, 85, 87, 3228]
2025-05-13 17:05:12,635 - INFO - Closing glyph list over 'GSUB': 44 glyphs before
2025-05-13 17:05:12,635 - INFO - Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'K', 'L', 'M', 'P', 'R', 'S', 'T', 'V', 'X', 'Z', 'a', 'b', 'd', 'e', 'f', 'five', 'four', 'g', 'hyphen', 'i', 'm', 'minus', 'n', 'nonmarkingreturn', 'one', 'r', 'six', 'space', 't', 'three', 'two', 'underscore', 'zero']
2025-05-13 17:05:12,636 - INFO - Glyph IDs:   [0, 1, 2, 3, 16, 19, 20, 21, 22, 23, 24, 25, 36, 37, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 51, 53, 54, 55, 57, 59, 61, 66, 68, 69, 71, 72, 73, 74, 76, 80, 81, 85, 87, 3228]
2025-05-13 17:05:12,640 - INFO - Closed glyph list over 'GSUB': 47 glyphs after
2025-05-13 17:05:12,647 - INFO - Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'K', 'L', 'M', 'P', 'R', 'S', 'T', 'V', 'X', 'Z', 'a', 'b', 'd', 'e', 'f', 'fi', 'five', 'four', 'g', 'hyphen', 'i', 'm', 'minus', 'n', 'nonmarkingreturn', 'one', 'r', 'six', 'space', 't', 'three', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
2025-05-13 17:05:12,647 - INFO - Glyph IDs:   [0, 1, 2, 3, 16, 19, 20, 21, 22, 23, 24, 25, 36, 37, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 51, 53, 54, 55, 57, 59, 61, 66, 68, 69, 71, 72, 73, 74, 76, 80, 81, 85, 87, 3228, 5038, 5039, 5041]
2025-05-13 17:05:12,648 - INFO - Closing glyph list over 'glyf': 47 glyphs before
2025-05-13 17:05:12,648 - INFO - Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'K', 'L', 'M', 'P', 'R', 'S', 'T', 'V', 'X', 'Z', 'a', 'b', 'd', 'e', 'f', 'fi', 'five', 'four', 'g', 'hyphen', 'i', 'm', 'minus', 'n', 'nonmarkingreturn', 'one', 'r', 'six', 'space', 't', 'three', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
2025-05-13 17:05:12,649 - INFO - Glyph IDs:   [0, 1, 2, 3, 16, 19, 20, 21, 22, 23, 24, 25, 36, 37, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 51, 53, 54, 55, 57, 59, 61, 66, 68, 69, 71, 72, 73, 74, 76, 80, 81, 85, 87, 3228, 5038, 5039, 5041]
2025-05-13 17:05:12,650 - INFO - Closed glyph list over 'glyf': 47 glyphs after
2025-05-13 17:05:12,650 - INFO - Glyph names: ['.notdef', '.null', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'K', 'L', 'M', 'P', 'R', 'S', 'T', 'V', 'X', 'Z', 'a', 'b', 'd', 'e', 'f', 'fi', 'five', 'four', 'g', 'hyphen', 'i', 'm', 'minus', 'n', 'nonmarkingreturn', 'one', 'r', 'six', 'space', 't', 'three', 'two', 'underscore', 'uniFB00', 'uniFB03', 'zero']
2025-05-13 17:05:12,651 - INFO - Glyph IDs:   [0, 1, 2, 3, 16, 19, 20, 21, 22, 23, 24, 25, 36, 37, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 51, 53, 54, 55, 57, 59, 61, 66, 68, 69, 71, 72, 73, 74, 76, 80, 81, 85, 87, 3228, 5038, 5039, 5041]
2025-05-13 17:05:12,652 - INFO - Retaining 47 glyphs
2025-05-13 17:05:12,659 - INFO - head subsetting not needed
2025-05-13 17:05:12,659 - INFO - hhea subsetting not needed
2025-05-13 17:05:12,660 - INFO - maxp subsetting not needed
2025-05-13 17:05:12,660 - INFO - OS/2 subsetting not needed
2025-05-13 17:05:12,664 - INFO - hmtx subsetted
2025-05-13 17:05:12,667 - INFO - cmap subsetted
2025-05-13 17:05:12,674 - INFO - fpgm subsetting not needed
2025-05-13 17:05:12,675 - INFO - prep subsetting not needed
2025-05-13 17:05:12,675 - INFO - cvt  subsetting not needed
2025-05-13 17:05:12,676 - INFO - loca subsetting not needed
2025-05-13 17:05:12,676 - INFO - post subsetted
2025-05-13 17:05:12,677 - INFO - gasp subsetting not needed
2025-05-13 17:05:12,677 - INFO - MATH subsetted
2025-05-13 17:05:12,680 - INFO - GDEF subsetted
2025-05-13 17:05:12,730 - INFO - GPOS subsetted
2025-05-13 17:05:12,733 - INFO - GSUB subsetted
2025-05-13 17:05:12,738 - INFO - name subsetting not needed
2025-05-13 17:05:12,741 - INFO - glyf subsetted
2025-05-13 17:05:12,742 - INFO - head pruned
2025-05-13 17:05:12,746 - INFO - OS/2 Unicode ranges pruned: [0, 38]
2025-05-13 17:05:12,747 - INFO - OS/2 CodePage ranges pruned: [0]
2025-05-13 17:05:12,748 - INFO - glyf pruned
2025-05-13 17:05:12,749 - INFO - GDEF pruned
2025-05-13 17:05:12,750 - INFO - GPOS pruned
2025-05-13 17:05:12,750 - INFO - GSUB pruned
2025-05-13 17:05:12,768 - INFO - name pruned
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/3_compute_pseudotime.html

**Contents:**
- scFates - Tree analysis¶
- Set up¶
- Functions¶
- Paths¶
- Import adata¶
- Actual time - kNN smoothing¶
- scVI¶
- Palantir¶
  - Pseudotime - scFates¶
- Compare real and pseudo-time¶
- Save object with UMAP, time and pseudotiem computed¶

ref: https://scfates.readthedocs.io/en/latest/Tree_Analysis_Bone_marrow_fates.html

**Examples:**

Example 1 (python):
```python
#Sets the 'R_HOME' environment variable to the location where Conda installed R libraries.
#You should run this code at the beginning of your Jupyter notebook, before importing any modules that rely on rpy2 or scFates.
#This will set the 'R_HOME' environment variable for the duration of your Python session
import os, sys
os.environ['R_HOME'] = sys.exec_prefix+"/lib/R/"
import scFates
```

Example 2 (python):
```python
import sys
!{sys.executable} -m pip -q install palantir fa2
```

Example 3 (python):
```python
from anndata import AnnData
import numpy as np
import pandas as pd
import scanpy as sc
import scFates as scf
import palantir
import matplotlib.pyplot as plt
import glob

sc.settings.verbosity = 3
sc.settings.logfile = sys.stdout
## fix palantir breaking down some plots
import seaborn
seaborn.reset_orig()
%matplotlib inline
import os

import warnings
warnings.filterwarnings('ignore')

sc.set_figure_params()
scf.set_figure_pubready()
```

Example 4 (python):
```python
import session_info
session_info.show()
```

Example 5 (python):
```python
-----
anndata             0.9.1
matplotlib          3.7.1
numpy               1.24.3
palantir            1.2
pandas              1.5.3
scFates             1.0.2
scanpy              1.9.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 6 (python):
```python
PIL                 9.5.0
adjustText          0.8
anyio               NA
arrow               1.2.3
asttokens           NA
attr                23.1.0
babel               2.12.1
backcall            0.2.0
backports           NA
certifi             2023.05.07
cffi                1.15.1
charset_normalizer  3.1.0
comm                0.1.3
cycler              0.10.0
cython_runtime      NA
dateutil            2.8.2
debugpy             1.6.7
decorator           5.1.1
defusedxml          0.7.1
executing           1.2.0
fastjsonschema      NA
fcsparser           0.2.6
fqdn                NA
h5py                3.8.0
idna                3.4
igraph              0.10.4
importlib_metadata  NA
importlib_resources NA
ipykernel           6.23.1
ipython_genutils    0.2.0
ipywidgets          8.0.6
isoduration         NA
jedi                0.18.2
jinja2              3.1.2
joblib              1.2.0
json5               NA
jsonpointer         2.3
jsonschema          4.17.3
jupyter_events      0.6.3
jupyter_server      2.6.0
jupyterlab_server   2.23.0
kiwisolver          1.4.4
leidenalg           0.9.1
llvmlite            0.40.0
markupsafe          2.1.2
matplotlib_inline   0.1.6
mpl_toolkits        NA
natsort             8.3.1
nbformat            5.9.0
networkx            2.8.8
numba               0.57.0
numexpr             2.8.4
overrides           NA
packaging           23.1
parso               0.8.3
patsy               0.5.3
pexpect             4.8.0
phenograph          1.5.7
pickleshare         0.7.5
pkg_resources       NA
platformdirs        3.5.1
plotly              5.14.1
progressbar         4.2.0
prometheus_client   NA
prompt_toolkit      3.0.38
psutil              5.9.5
ptyprocess          0.7.0
pure_eval           0.2.2
pydev_ipython       NA
pydevconsole        NA
pydevd              2.9.5
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygam               0.8.0
pygments            2.15.1
pyparsing           3.0.9
pyrsistent          NA
python_utils        NA
pythonjsonlogger    NA
pytz                2023.3
requests            2.31.0
rfc3339_validator   0.1.4
rfc3986_validator   0.1.1
rpy2                3.5.11
scipy               1.10.1
send2trash          NA
setuptools          67.7.2
simpleppt           1.1.3
six                 1.16.0
sklearn             1.2.2
skmisc              0.2.0
sniffio             1.3.0
stack_data          0.6.2
statsmodels         0.14.0
texttable           1.6.7
threadpoolctl       3.1.0
tornado             6.3.2
tqdm                4.65.0
traitlets           5.9.0
typing_extensions   NA
tzlocal             NA
uri_template        NA
urllib3             2.0.3
wcwidth             0.2.6
webcolors           1.13
websocket           1.5.2
yaml                6.0
zipp                NA
zmq                 25.1.0
```

Example 7 (unknown):
```unknown
-----
IPython             8.12.2
jupyter_client      8.2.0
jupyter_core        5.3.0
jupyterlab          4.0.2
notebook            6.5.4
-----
Python 3.8.16 | packaged by conda-forge | (default, Feb  1 2023, 16:01:55) [GCC 11.3.0]
Linux-5.4.0-162-generic-x86_64-with-glibc2.10
-----
Session information updated at 2023-10-24 21:11
```

Example 8 (python):
```python
sc.settings.verbosity = 3             # verbosity: errors (0), warnings (1), info (2), hints (3)
sc.settings.set_figure_params(dpi=80, facecolor='white', color_map = 'RdPu',figsize=(4,4))
```

Example 9 (python):
```python
from datetime import datetime

today = datetime.today().strftime('%Y-%m-%d')
print(today)
```

Example 10 (python):
```python
def plot_and_store(init_pos, adata):
    """
    This function selects the columns of the obsm layer specified by init_pos, 
    draws a graph using these coordinates as the initial positions, and stores the graph 
    coordinates in a new obsm layer named 'X_draw_graph_fa_' followed by the value of init_pos. 
    The modified AnnData object is then returned.

    If adata.obsm[f'X_draw_graph_fa_{init_pos}'] already exists, it puts that in adata.obsm['X_draw_graph_fa'].
    If adata.obsm[f'X_draw_graph_fa_{init_pos}'] does not exist, it performs sc.tl.draw_graph() as before.

    Parameters
    ----------
    init_pos : str
        Initial position for the graph. Should be in the format 'X_palantir12'.
    
    adata : AnnData
        The annotated data matrix of shape n_obs x n_vars. Rows correspond
        to cells and columns to genes.

    Returns
    -------
    adata : AnnData
        The modified annotated data matrix.
    """
    x_col = int(init_pos[-2:][0]) - 1
    y_col = int(init_pos[-2:][1]) - 1
    selected_cols = [x_col, y_col]
    adata.obsm[f'{init_pos}'] = adata.obsm[f'{init_pos[:-2]}'][:,selected_cols]
    
    if f'X_draw_graph_fa_{init_pos}' in adata.obsm:
        print(f'graph using {init_pos} has previously been calculated, retrieving that...')
        adata.obsm['X_draw_graph_fa'] = adata.obsm[f'X_draw_graph_fa_{init_pos}']
    else:
        print('calculating graph coordinates...')
        sc.tl.draw_graph(adata,init_pos=init_pos)
        adata.obsm[f'X_draw_graph_fa_{init_pos}'] = adata.obsm['X_draw_graph_fa'].copy()
    return adata
```

Example 11 (python):
```python
def plot_embedding(
    adata,
    groupby,
    basis="umap",
    color=None,
    annot=True,
    min_group_size=0,
    greyout_group=None,
    highlight_group=None,
    size=None,
    use_uns_colors=True,
    save=None,
    savedpi=300,
    figsize=(4, 4),
    **kwargs,
):
    set_figsize(figsize)
    if f"X_{basis}" not in adata.obsm.keys():
        raise KeyError(f'"X_{basis}" not found in `adata.obsm`.')
    if min_group_size:
        group_sizes = adata.obs[groupby].value_counts()
        good_groups = group_sizes[group_sizes >= min_group_size].index
        adata = adata[adata.obs[groupby].isin(good_groups)].copy()
    if isinstance(groupby, (list, tuple)):
        groupby = groupby[0]
    if groupby not in adata.obs.columns:
        raise KeyError(f'"{groupby}" not found in `adata.obs`.')
    if adata.obs[groupby].dtype.name != "category":
        if (
            isinstance(adata.obs[groupby][0], (str, bool, np.bool_))
            and adata.obs[groupby].unique().size < 100
        ):
            adata.obs[groupby] = adata.obs[groupby].astype(str).astype("category")
        else:
            sc_warn(f'"{groupby}" is not categorical.')
            plot_scatter(adata, basis=basis, color=groupby, **kwargs)
    categories = list(adata.obs[groupby].cat.categories)
    rename_dict1 = {
        ct: f"{i:^5d} {ct} (n={(adata.obs[groupby]==ct).sum()})" for i, ct in enumerate(categories)
    }
    restore_dict1 = {
        f"{i:^5d} {ct} (n={(adata.obs[groupby]==ct).sum()})": ct for i, ct in enumerate(categories)
    }
    rename_dict2 = {ct: f"{i:^5d} {ct}" for i, ct in enumerate(categories)}
    restore_dict2 = {f"{i:^5d} {ct}": ct for i, ct in enumerate(categories)}

    marker_size = size
    kwargs["show"] = False
    kwargs["save"] = False
    kwargs["frameon"] = kwargs.get("frameon", None)
    kwargs["legend_loc"] = kwargs.get("legend_loc", "right margin")

    if color is not None and color != groupby:
        use_uns_colors = False
    color = groupby if color is None else color
    offset = 0 if "diffmap" in basis else -1
    xi, yi = 1, 2
    if "components" in kwargs:
        xi, yi = kwargs["components"]
    xi += offset
    yi += offset

    if not (use_uns_colors and f"{color}_colors" in adata.uns):
        adata.uns[f"{color}_colors"] = make_palette(
            adata.obs[color].cat.categories.size, kwargs.get("palette", None)
        )
        if greyout_group is not None:
            if isinstance(greyout_group, str):
                greyout_group = [greyout_group]
            greyout_indices = np.where(
                adata.obs[color].cat.categories.isin(greyout_group)
            )[0]
            for nan_i in greyout_indices:
                adata.uns[f"{color}_colors"][nan_i] = "#E0E0E0"

        if highlight_group is not None:
            if isinstance(highlight_group, str):
                highlight_group = [highlight_group]
            nohl_indices = np.where(
                ~adata.obs[color].cat.categories.isin(highlight_group)
            )[0]
            for nohl_i in nohl_indices:
                adata.uns[f"{color}_colors"][nohl_i] = "#E0E0E0"

    if annot == "full":
        adata.obs[groupby].cat.rename_categories(rename_dict1, inplace=True)
    elif annot in (None, False, "none"):
        kwargs["title"] = ""
        kwargs["legend_loc"] = None
    else:
        adata.obs[groupby].cat.rename_categories(rename_dict2, inplace=True)

    try:
        if "ax" in kwargs:
            ax = kwargs["ax"]
            plot_scatter(adata, basis=basis, color=color, size=marker_size, **kwargs)
        else:
            ax = plot_scatter(adata, basis=basis, color=color, size=marker_size, **kwargs)
    finally:
        if annot == "full":
            adata.obs[groupby].cat.rename_categories(restore_dict1, inplace=True)
        elif annot not in (None, False, "none"):
            adata.obs[groupby].cat.rename_categories(restore_dict2, inplace=True)
    if annot not in (None, False, "none"):
        centroids = pseudo_bulk(adata, groupby, use_rep=f"X_{basis}", FUN=np.median).T
        fontsize = kwargs["legend_fontsize"] if "legend_fontsize" in kwargs else 11
        texts = [
            ax.text(x=row[xi], y=row[yi], s=f"{i:d}", fontsize=fontsize - 1.5, fontweight="bold")
            for i, row in centroids.reset_index(drop=True).iterrows()
            if row[0].astype(str) != "nan"
        ]
        from adjustText import adjust_text

        adjust_text(texts, ax=ax, text_from_points=False, autoalign=False)
        version_scale = 1 if sc.__version__.startswith("1.4") else 1
        ax.legend(
            loc="center left",
            bbox_to_anchor=(1, 0.5),
            frameon=False,
            ncol=int(np.ceil(len(categories) / int(figsize[1] * 3) / 8 * fontsize)),
            fontsize=fontsize * 1.1,
            markerscale=3 * (fontsize / 11) * version_scale,
            handletextpad=-1.9,
            labelspacing=9 / fontsize,
        )
    if save:
        plt.savefig(fname=save, dpi=savedpi, bbox_inches="tight", pad_inches=0.1)
    if "ax" not in kwargs:
        return ax
```

Example 12 (python):
```python
# Parameters
job_name = "aCMs_WholeSample"
n_layers = 1
developing = False
paired_adata_path = "/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA/aCMs_WholeSample_paired.h5ad"
unimodality_adata_path = "none"
combination = "multivi_aCMs_WholeSample_n_layers1"


latent_variables_dir=os.path.join("/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA","latent_variables") # for latent variables
latent_variables_path=os.path.join(latent_variables_dir, f"mvi_{combination}_latent_variables.h5ad")
```

Example 13 (python):
```python
adata = sc.read(paired_adata_path)
adata
```

Example 14 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
```

Example 15 (python):
```python
latent_variables = sc.read(latent_variables_path)
latent_variables
```

Example 16 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 20
```

Example 17 (python):
```python
latent_variables.shape
```

Example 18 (python):
```python
adata.obsm["X_multiVI"]=latent_variables.X
adata
```

Example 19 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    obsm: 'X_multiVI'
```

Example 20 (python):
```python
sc.pp.neighbors(adata,use_rep="X_multiVI")
```

Example 21 (unknown):
```unknown
computing neighbors
    finished: added to `.uns['neighbors']`
    `.obsp['distances']`, distances for each pair of neighbors
    `.obsp['connectivities']`, weighted adjacency matrix (0:00:07)
```

Example 22 (python):
```python
sc.tl.umap(adata,min_dist=0.01,n_components=3)
```

Example 23 (python):
```python
computing UMAP
    finished: added
    'X_umap', UMAP coordinates (adata.obsm) (0:00:11)
```

Example 24 (python):
```python
sc.pl.umap(adata,color=["fine_grain","week"])
```

Example 25 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'neighbors', 'umap', 'fine_grain_colors', 'week_colors'
    obsm: 'X_multiVI', 'X_umap'
    obsp: 'distances', 'connectivities'
```

Example 26 (python):
```python
# Function to convert weeks to days
def week_to_days(week_str):
    # Extract the numerical part and convert to integer
    week_num = int(week_str.replace('W', ''))
    return week_num * 7

# Create a new column 'age_days' in adata.obs DataFrame
adata.obs['age_days'] = adata.obs['week'].apply(week_to_days)
adata.obs['age_days'] = adata.obs['age_days'].astype(float)
adata.obs['age_days'].value_counts()
```

Example 27 (unknown):
```unknown
91.0    3925
35.0     443
63.0     417
84.0     372
56.0     321
49.0     186
Name: age_days, dtype: int64
```

Example 28 (python):
```python
sc.pl.umap(adata,color=["fine_grain","donor","age_days"],cmap='viridis',wspace=0.5)
```

Example 29 (python):
```python
import scipy.sparse

column='age_days'

# Extract the key for the connectivities matrix
conn_key = adata.uns['neighbors']['connectivities_key']

# Now get the actual connectivities matrix using this key
conn_matrix = adata.obsp[conn_key]

# Convert age_days to a NumPy array
age_days = adata.obs[column].values

# Calculate the weighted sum of neighbors for each node
weighted_sum = conn_matrix.dot(age_days)

# Calculate the sum of weights for each node (including self)
sum_weights = conn_matrix.sum(axis=1).A1  # A1 returns a flattened ndarray

# Calculate the weighted average
age_days_smoothed = weighted_sum / sum_weights

# Add the smoothed values as a new column
adata.obs[f'{column}_smoothed_once'] = age_days_smoothed
```

Example 30 (python):
```python
# Min-max normalization
min_val = adata.obs['age_days'].min()
max_val = adata.obs['age_days'].max()
adata.obs['age_days_normalized'] = (adata.obs['age_days'] - min_val) / (max_val - min_val)
```

Example 31 (python):
```python
sc.pl.umap(adata,color=["fine_grain","age_days","age_days_normalized"],cmap='rainbow',wspace=0.5)
```

Example 32 (python):
```python
# Function to perform iterative smoothing until a stable state is achieved
def iterative_smoothing(adata, feature_col, tol=1e-10, max_iter=1000):
    # Extract the key for the connectivities matrix
    conn_key = adata.uns['neighbors']['connectivities_key']

    # Get the actual connectivities matrix using this key
    conn_matrix = adata.obsp[conn_key]

    # Convert feature to a NumPy array
    feature_values = adata.obs[feature_col].values

    # Initialize variables to keep track of previous and current state
    prev_values = feature_values.copy()
    iter_count = 0

    while iter_count < max_iter:
        # Calculate the weighted sum of neighbors for each node
        weighted_sum = conn_matrix.dot(feature_values)

        # Calculate the sum of weights for each node (including self)
        sum_weights = conn_matrix.sum(axis=1).A1  # A1 returns a flattened ndarray

        # Calculate the new feature values (weighted average)
        feature_values = weighted_sum / sum_weights

        # Check for convergence (within given tolerance)
        diff = np.abs(feature_values - prev_values).max()
        if diff < tol:
            print(f"Converged after {iter_count} iterations.")
            break

        # Update previous values for the next iteration
        prev_values = feature_values.copy()

        # Increment iteration count
        iter_count += 1

    # If maximum iterations reached
    if iter_count == max_iter:
        print(f"Max iterations reached. Last diff: {diff}")

    # Add the smoothed values as a new column
    new_col_name = f"{feature_col}_smoothed_iteratively"
    adata.obs[new_col_name] = feature_values
```

Example 33 (python):
```python
# Run the function

# Function to perform one round of smoothing on a given column of an AnnData object
def smooth_column_once(adata, column):
    """
    Smooths a given column in an AnnData object's obs DataFrame based on neighborhood graph.
    
    Parameters:
        adata (anndata.AnnData): The AnnData object containing the neighborhood graph and data.
        column (str): The column in the obs DataFrame to be smoothed.
    
    Returns:
        None: The AnnData object's obs DataFrame is modified in-place with the smoothed column.
    """
    # Extract the key for the connectivities matrix
    conn_key = adata.uns['neighbors']['connectivities_key']
    
    # Get the actual connectivities matrix using this key
    conn_matrix = adata.obsp[conn_key]
    
    # Convert the column values to a NumPy array
    column_values = adata.obs[column].values
    
    # Calculate the weighted sum of neighbors for each node
    weighted_sum = conn_matrix.dot(column_values)
    
    # Calculate the sum of weights for each node (including self)
    sum_weights = conn_matrix.sum(axis=1).A1  # A1 returns a flattened ndarray
    
    # Calculate the weighted average
    smoothed_values = weighted_sum / sum_weights
    
    # Add the smoothed values as a new column
    new_col_name = f"{column}_smoothed_once"
    adata.obs[new_col_name] = smoothed_values
```

Example 34 (python):
```python
smooth_column_once(adata,'age_days')
smooth_column_once(adata,'age_days_normalized')

tol=1e-6
max_iter=100000

iterative_smoothing(adata, 'age_days', tol=tol, max_iter=max_iter)
iterative_smoothing(adata, 'age_days_normalized', tol=1e-6, max_iter=max_iter)
```

Example 35 (unknown):
```unknown
Converged after 608 iterations.
Converged after 436 iterations.
```

Example 36 (python):
```python
sc.pl.umap(adata,color=["age_days","age_days_smoothed_once","age_days_smoothed_iteratively",
                        "age_days_normalized","age_days_normalized_smoothed_once","age_days_normalized_smoothed_iteratively",
                        "fine_grain"],
           cmap='rainbow',wspace=0.5,ncols=3)
```

Example 37 (python):
```python
sc.pl.umap(adata,color=["TBX18","SHOX2","NTM","PANCR"],cmap='RdPu',wspace=0.5,ncols=2)
```

Example 38 (unknown):
```unknown
temporal_ordering="age_days_smoothed_iteratively"
```

Example 39 (python):
```python
global_adata = sc.read('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw.h5ad')
global_adata
```

Example 40 (unknown):
```unknown
AnnData object with n_obs × n_vars = 297473 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'mid_grain_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 41 (python):
```python
# Grab scVI latent variables for just the cells in adata
adata.obsm["X_scVI"] = global_adata[global_adata.obs.index.isin(adata.obs.index)].obsm["X_scVI"].copy()
```

Example 42 (python):
```python
# projectsions from scVI
projections = pd.DataFrame(adata.obsm["X_scVI"],index=adata.obs_names)
```

Example 43 (unknown):
```unknown
# fate probability mapping of the cells
dm_res = palantir.utils.run_diffusion_maps(projections)
ms_data = palantir.utils.determine_multiscale_space(dm_res,n_eigs=4)
```

Example 44 (unknown):
```unknown
Determing nearest neighbor graph...
computing neighbors
    finished: added to `.uns['neighbors']`
    `.obsp['distances']`, distances for each pair of neighbors
    `.obsp['connectivities']`, weighted adjacency matrix (0:00:01)
```

Example 45 (python):
```python
# Generate embedding from the multiscale diffusion space
adata.obsm["X_palantirscVI"]=ms_data.values
```

Example 46 (python):
```python
def plot_and_store(init_pos, adata):
    """
    This function selects the columns of the obsm layer specified by init_pos, 
    draws a graph using these coordinates as the initial positions, and stores the graph 
    coordinates in a new obsm layer named 'X_draw_graph_fa_' followed by the value of init_pos. 
    The modified AnnData object is then returned.

    If adata.obsm[f'X_draw_graph_fa_{init_pos}'] already exists, it puts that in adata.obsm['X_draw_graph_fa'].
    If adata.obsm[f'X_draw_graph_fa_{init_pos}'] does not exist, it performs sc.tl.draw_graph() as before.

    Parameters
    ----------
    init_pos : str
        Initial position for the graph. Should be in the format 'X_palantir12'.
    
    adata : AnnData
        The annotated data matrix of shape n_obs x n_vars. Rows correspond
        to cells and columns to genes.

    Returns
    -------
    adata : AnnData
        The modified annotated data matrix.
    """
    x_col = int(init_pos[-2:][0]) - 1
    y_col = int(init_pos[-2:][1]) - 1
    selected_cols = [x_col, y_col]
    adata.obsm[f'{init_pos}'] = adata.obsm[f'{init_pos[:-2]}'][:,selected_cols]
    
    if f'X_draw_graph_fa_{init_pos}' in adata.obsm:
        print(f'graph using {init_pos} has previously been calculated, retrieving that...')
        adata.obsm['X_draw_graph_fa'] = adata.obsm[f'X_draw_graph_fa_{init_pos}']
    else:
        print('calculating graph coordinates...')
        sc.tl.draw_graph(adata,init_pos=init_pos)
        adata.obsm[f'X_draw_graph_fa_{init_pos}'] = adata.obsm['X_draw_graph_fa'].copy()
    return adata
```

Example 47 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'neighbors', 'umap', 'fine_grain_colors', 'week_colors', 'donor_colors'
    obsm: 'X_multiVI', 'X_umap', 'X_scVI', 'X_palantir'
    obsp: 'distances', 'connectivities'
```

Example 48 (python):
```python
%%time
embeddings=['X_umap12','X_umap13','X_palantir12','X_palantir13', 'X_scVI12','X_scVI13','X_palantirscVI12','X_palantirscVI13']

for embedding in embeddings:
        print(embedding)
        adata = plot_and_store(embedding, adata)
        sc.pl.draw_graph(adata,color=['fine_grain','age_days'],wspace=0.8, ncols=2,cmap='bwr', alpha=0.8)
```

Example 49 (unknown):
```unknown
X_umap12
graph using X_umap12 has previously been calculated, retrieving that...
```

Example 50 (unknown):
```unknown
X_umap13
graph using X_umap13 has previously been calculated, retrieving that...
```

Example 51 (unknown):
```unknown
X_palantir12
graph using X_palantir12 has previously been calculated, retrieving that...
```

Example 52 (unknown):
```unknown
X_palantir13
graph using X_palantir13 has previously been calculated, retrieving that...
```

Example 53 (unknown):
```unknown
X_scVI12
graph using X_scVI12 has previously been calculated, retrieving that...
```

Example 54 (unknown):
```unknown
X_scVI13
graph using X_scVI13 has previously been calculated, retrieving that...
```

Example 55 (python):
```python
X_palantirscVI12
calculating graph coordinates...
drawing single-cell graph using layout 'fa'
    finished: added
    'X_draw_graph_fa', graph_drawing coordinates (adata.obsm) (0:00:34)
```

Example 56 (python):
```python
X_palantirscVI13
calculating graph coordinates...
drawing single-cell graph using layout 'fa'
    finished: added
    'X_draw_graph_fa', graph_drawing coordinates (adata.obsm) (0:00:33)
```

Example 57 (unknown):
```unknown
CPU times: user 1min 4s, sys: 3.06 s, total: 1min 7s
Wall time: 1min 15s
```

Example 58 (python):
```python
scf.tl.tree(adata,method="ppt",Nodes=50,use_rep="umap",
            device="cpu",seed=1,ppt_lambda=10000,ppt_sigma=0.4,ppt_nsteps=200)

scf.pl.graph(adata)
```

Example 59 (unknown):
```unknown
inferring a principal tree --> parameters used 
    50 principal points, sigma = 0.4, lambda = 10000, metric = euclidean
    fitting:   5%|█████████▎                                                                                                                                                                                 | 10/200 [00:01<00:21,  8.81it/s]
    converged
    finished (0:00:01) --> added 
    .uns['ppt'], dictionnary containing inferred tree.
    .obsm['X_R'] soft assignment of cells to principal points.
    .uns['graph']['B'] adjacency matrix of the principal points.
    .uns['graph']['F'] coordinates of principal points in representation space.
```

Example 60 (python):
```python
scf.tl.root(adata,49)
```

Example 61 (unknown):
```unknown
node 49 selected as a root --> added
    .uns['graph']['root'] selected root.
    .uns['graph']['pp_info'] for each PP, its distance vs root and segment assignment.
    .uns['graph']['pp_seg'] segments network information.
```

Example 62 (python):
```python
scf.tl.pseudotime(adata,n_jobs=20,n_map=100,seed=42)
```

Example 63 (unknown):
```unknown
projecting cells onto the principal graph
    mappings:   0%|                                                                                                                                                               | 0/100 [00:00<?, ?it/s]
```

Example 64 (unknown):
```unknown
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
Unable to determine R library path: Command '('/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/scfates_env/lib/R/bin/Rscript', '-e', 'cat(Sys.getenv("LD_LIBRARY_PATH"))')' returned non-zero exit status 2.
```

Example 65 (python):
```python
mappings: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:51<00:00,  1.93it/s]
    finished (0:00:57) --> added
    .obs['edge'] assigned edge.
    .obs['t'] pseudotime value.
    .obs['seg'] segment of the tree assigned.
    .obs['milestones'] milestone assigned.
    .uns['pseudotime_list'] list of cell projection from all mappings.
```

Example 66 (python):
```python
scf.pl.trajectory(adata)
```

Example 67 (unknown):
```unknown
temporal_ordering='age_days_smoothed_once'
```

Example 68 (python):
```python
sc.pl.umap(adata,color=["seg","milestones","t",temporal_ordering],ncols=2)
```

Example 69 (python):
```python
import scipy.stats

# Assuming `adata` is your AnnData object and it has 't' and 'x' in its obs DataFrame
t_values = adata.obs['t']
x_values = adata.obs[temporal_ordering]

# Calculate the Pearson correlation coefficient and the p-value
correlation_coefficient, p_value = scipy.stats.pearsonr(t_values, x_values)

plt.scatter(t_values,x_values)
correlation_coefficient, p_value
```

Example 70 (unknown):
```unknown
(0.8275538981212454, 0.0)
```

Example 71 (python):
```python
import seaborn as sns
sns.histplot(adata.obs.age_days)
```

Example 72 (unknown):
```unknown
<Axes: xlabel='age_days', ylabel='Count'>
```

Example 73 (python):
```python
sns.histplot(adata.obs[temporal_ordering])
```

Example 74 (unknown):
```unknown
<Axes: xlabel='age_days_smoothed_once', ylabel='Count'>
```

Example 75 (python):
```python
sns.histplot(adata.obs['t'])
```

Example 76 (unknown):
```unknown
<Axes: xlabel='t', ylabel='Count'>
```

Example 77 (python):
```python
paired_adata_with_time_path = "/lustre/scratch126/cellgen/team205/heart/objects/fetal/RNA/aCMs_WholeSample_paired_with_time.h5ad"
adata.write(paired_adata_with_time_path)
adata
```

Example 78 (unknown):
```unknown
AnnData object with n_obs × n_vars = 5664 × 544641
    obs: 'sangerID', 'combinedID', 'donor', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'multiplexed', 'batch_key', 'region', 'FACSgate', 'sex', 'week', 'trimester', 'fine_grain', 'mid_grain', 'coarse_grain', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', 'stress_score', 'hb1_score', 'age_days', 'age_days_smoothed_once', 'age_days_normalized', 'age_days_normalized_smoothed_once', 'age_days_smoothed_iteratively', 'age_days_normalized_smoothed_iteratively', 't', 'seg', 'edge', 't_sd', 'milestones'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id'
    uns: 'neighbors', 'umap', 'fine_grain_colors', 'week_colors', 'donor_colors', 'graph', 'ppt', 'pseudotime_list', 'milestones_colors', 'seg_colors'
    obsm: 'X_multiVI', 'X_umap', 'X_R'
    obsp: 'distances', 'connectivities'
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/TransmuralAxis/tm_axis_nuclei-density.html

**Contents:**
- Import modules¶
- Functions¶
- Read in adata¶
- Add axis¶
- Load calculated image features and add to anndata¶
- Plot¶

**Examples:**

Example 1 (python):
```python
import os
import numpy as np
import scanpy as sc
import anndata
import pandas as pd
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 
import matplotlib.pyplot as plt
import seaborn as sns

import squidpy as sq

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter("ignore", UserWarning)
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
anndata             0.9.2
matplotlib          3.7.2
numpy               1.22.4
pandas              2.0.3
scanpy              1.9.4
seaborn             0.12.2
session_info        1.0.0
squidpy             1.2.3
-----
```

Example 4 (python):
```python
PIL                 10.0.0
asciitree           NA
asttokens           NA
backcall            0.2.0
cloudpickle         2.2.1
comm                0.1.4
cycler              0.10.0
cython_runtime      NA
dask                2023.5.0
dask_image          2023.03.0
dateutil            2.8.2
debugpy             1.7.0
decorator           5.1.1
docrep              0.3.2
entrypoints         0.4
executing           1.2.0
fasteners           0.18
h5py                3.9.0
igraph              0.10.6
imagecodecs         2023.3.16
imageio             2.31.3
importlib_metadata  NA
importlib_resources NA
ipykernel           6.25.2
jedi                0.19.0
jinja2              3.1.2
joblib              1.3.2
kiwisolver          1.4.5
lazy_loader         NA
leidenalg           0.10.1
llvmlite            0.38.1
markupsafe          2.1.3
matplotlib_scalebar 0.8.1
mpl_toolkits        NA
mpmath              1.3.0
natsort             8.4.0
networkx            3.1
numba               0.55.2
numcodecs           0.11.0
nvfuser             NA
packaging           23.1
parso               0.8.3
patsy               0.5.3
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
platformdirs        3.10.0
prompt_toolkit      3.0.39
psutil              5.9.5
ptyprocess          0.7.0
pure_eval           0.2.2
pydev_ipython       NA
pydevconsole        NA
pydevd              2.9.5
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.16.1
pyparsing           3.0.9
pytz                2023.3.post1
pywt                1.4.1
scipy               1.10.1
six                 1.16.0
skimage             0.21.0
sklearn             1.3.0
stack_data          0.6.2
statsmodels         0.14.0
sympy               1.12
texttable           1.6.7
threadpoolctl       3.2.0
tifffile            2023.7.10
tlz                 0.12.0
toolz               0.12.0
torch               2.0.1+cu117
tornado             6.3.3
tqdm                4.66.1
traitlets           5.9.0
typing_extensions   NA
validators          0.22.0
wcwidth             0.2.6
xarray              2023.1.0
yaml                6.0.1
zarr                2.16.1
zipp                NA
zmq                 25.1.1
```

Example 5 (unknown):
```unknown
-----
IPython             8.12.2
jupyter_client      8.3.1
jupyter_core        5.3.1
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-5.4.0-169-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-02-13 15:12
```

Example 6 (python):
```python
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 7 (python):
```python
import importlib.util
import sys

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/tissueaxis.py")
tissueaxis = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = tissueaxis
spec.loader.exec_module(tissueaxis)
```

Example 8 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 9 (python):
```python
# This function selects the data for multiple slides from the spatial anndata object.
# Modified `select_slide` in cell2loc for multiple slides

# @param adata: anndata object with multiple spatial experiments
# @param s: list of selected slides
# @param s_col: column in adata.obs listing experiment name for each location

def select_multislide(adata, s, s_col='sample'):

    slides = adata[adata.obs[s_col].isin(s), :]
    s_keys = list(slides.uns['spatial'].keys())
    
    uns_dict = {}
    for ss in s:
        s_spatial=np.array(s_keys)[[ss in k for k in s_keys]][0]
        uns_dict[s_spatial]=slides.uns['spatial'][s_spatial]
    
    slides.uns['spatial'] = uns_dict

    return slides
```

Example 10 (python):
```python
adata_oct_healthy = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT_foetal_raw.h5ad')
print(adata_oct_healthy.shape)
adata_oct_healthy_LV = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT-LV_foetal_raw.h5ad')
print(adata_oct_healthy_LV.shape)
adata_oct_t21 = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT-T21_foetal_raw.h5ad')
print(adata_oct_t21.shape)
```

Example 11 (unknown):
```unknown
(14261, 33538)
(6767, 33538)
(11238, 33538)
```

Example 12 (python):
```python
# concatenate oct adata
adata_oct_healthy_LV.obs['status'] = 'Healthy'
adata_oct_t21.obs['status'] = 'Trisomy21'

adata_oct = adata_oct_healthy.concatenate(
    [adata_oct_healthy_LV,adata_oct_t21],
    batch_key=None,
    index_unique=None,
    join="inner"
)

adata_oct.obs['status'].value_counts()
```

Example 13 (unknown):
```unknown
status
Healthy      21028
Trisomy21    11238
Name: count, dtype: int64
```

Example 14 (python):
```python
# add
oct_samples = list(set(adata_oct.obs['sample']))

adata_oct.obs['transmural_axis'] = np.nan

axis_files = os.listdir('/nfs/team205/heart/visium_axis/foetal/transmural_LV')
for file in axis_files:
    # read in
    axis = pd.read_csv(f'/nfs/team205/heart/visium_axis/foetal/transmural_LV/{file}',index_col=0)
    # add to anndata
    if file.split('.csv')[0] in oct_samples:
        adata_oct.obs.loc[axis.index,'transmural_axis'] = axis.copy()

# select spots with axis
adata_oct = adata_oct[adata_oct.obs['transmural_axis'].isna()==False]
```

Example 15 (python):
```python
# remove nuclei segmentation failed samples: 20PCW samples
adata_oct = adata_oct[adata_oct.obs['sample'].isin(['HCAHeartST13162330','HCAHeartST13162331'])==False]
```

Example 16 (unknown):
```unknown
!ls /nfs/team205/heart/visium_axis/foetal/nuclei_segmentation/calculated_img_features
```

Example 17 (unknown):
```unknown
BHF_F_Hea13295830_img_features.csv   HCAHeartST11605167_img_features.csv
BHF_F_Hea13295831_img_features.csv   HCAHeartST11605168_img_features.csv
HCAHeartST10317184_img_features.csv  HCAHeartST13162330_img_features.csv
HCAHeartST10317185_img_features.csv  HCAHeartST13162331_img_features.csv
HCAHeartST10317186_img_features.csv  HCAHeartST13162336_img_features.csv
HCAHeartST11605165_img_features.csv
```

Example 18 (unknown):
```unknown
path_to_calculated_img_features = '/nfs/team205/heart/visium_axis/foetal/nuclei_segmentation/calculated_img_features'
img_feature_files = os.listdir(path_to_calculated_img_features)
len(img_feature_files)
```

Example 19 (python):
```python
for file in img_feature_files:
    # read in
    features = pd.read_csv(f'{path_to_calculated_img_features}/{file}',index_col=0)
    # add
    if file.split('_img')[0] in ['BHF_F_Hea13295830','BHF_F_Hea13295831']:
        continue
    else:
        cell_ids = list(set(features.index).intersection(adata_oct.obs_names))
        adata_oct.obs.loc[cell_ids,features.columns] = features

# checking whether all cells have feature
print(sum(adata_oct.obs['segmentation_label'].isna())==0)
```

Example 20 (unknown):
```unknown
features.head()
```

Example 21 (python):
```python
plt.rcParams['figure.figsize'] = 4.5,2
adata_oct.obs['segmentation_label'].hist(bins=100)
```

Example 22 (unknown):
```unknown
## remove most outer and inner bins, since those are not myocardium
# ad = ad[ad.obs["transmural_axis_bin"]!=ad.obs['transmural_axis_bin'].max()]
# ad = ad[ad.obs["transmural_axis_bin"]!=ad.obs['transmural_axis_bin'].min()]

# multiple by two for the FFPE samples since the thickness is 10um whereas OCT samples are 5um
# cell_id = ad.obs_names[ad.obs['cell2loc-map_group']=='16PCW_FFPE']
# ad.obs.loc[cell_id,'segmentation_label'] = ad.obs.loc[cell_id,'segmentation_label']*2
```

Example 23 (python):
```python
adata_all.obs['cell2loc-map_group'].value_counts()
```

Example 24 (unknown):
```unknown
cell2loc-map_group
16PCW_FFPE    4605
T21_13PCW     1940
13PCW         1837
7PCW           341
5PCW           293
Name: count, dtype: int64
```

Example 25 (python):
```python
tissueaxis.plot_axis_split(adata_all,
                           feature_to_plot='segmentation_label',
                           feature_type='obs',
                           min_max_scale=False,
                           splitby='cell2loc-map_group',
                           groups_order=[ '5PCW', '7PCW','13PCW','T21_13PCW'],
                           figsize=(5, 4),
                           cbar_pos=(0, 0.7, 0.05, 0.18),
                           cmap='Reds',
                           # save=f'{figdir}/tm-axis_{features_to_plot}_per-week.pdf'
                          )
```

Example 26 (python):
```python
tissueaxis.plot_axis_split(adata_all,
                           feature_to_plot='segmentation_label',
                           feature_type='obs',
                           min_max_scale=False,
                           splitby='cell2loc-map_group',
                           groups_order=[ '5PCW', '7PCW','13PCW'],
                           figsize=(5, 3.25),
                           cbar_pos=(0, 0.7, 0.03, 0.18),
                           cmap='Reds',
                           save=f'{figdir}/tm-axis_nuclei-segmentation_per-week.pdf'
                          )
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/Visium_mapping/make_aCMimmature_annotation_for_c2l.html

**Contents:**
- Make custom annotation for cell2location¶

A custom annotation which is the same as fine_grain except for the addition of "AtrialCardiomycoytesImmature"

The idea is to inspect whhether this signature is seen in the eary visium samples.

5664 rows × 1 columns

**Examples:**

Example 1 (python):
```python
import pandas as pd
import os
import scanpy as sc
```

Example 2 (python):
```python
# read in custom annotation (only applied to aCMs of course)
annotation_df=pd.read_csv("AtrialCardiomyocytesImmature_annotation.csv",index_col=0)
annotation_df
```

Example 3 (python):
```python
# read in the main object
adata=sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/Feb28ObjectRaw.h5ad')
adata
```

Example 4 (unknown):
```unknown
AnnData object with n_obs × n_vars = 297473 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'region', 'age', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet', 'batch_key', '_scvi_batch', 'FACSgate', 'fine_grain', 'mid_grain', 'coarse_grain', 'sex', 'week', 'trimester', 'heart_or_greatvessels', 'cycling', 'S_score', 'G2M_score', 'phase', '_scvi_labels', 'stress_score', 'hb1_score'
    var: 'gene_name_scRNA-0', 'gene_id'
    uns: 'FACSgate_colors', '_scvi_manager_uuid', '_scvi_uuid', 'age_colors', 'cell_or_nuclei_colors', 'coarse_grain_colors', 'cycling_colors', 'donor_colors', 'fine_grain_colors', 'heart_or_greatvessels_colors', 'hvg', 'kit_10x_colors', 'mid_grain_colors', 'modality_colors', 'neighbors', 'phase_colors', 'region_colors', 'sex_colors', 'trimester_colors', 'umap', 'week_colors'
    obsm: 'X_scVI', 'X_umap', '_scvi_extra_categorical_covs', '_scvi_extra_continuous', '_scvi_extra_continuous_covs', 'latent_gene_encoding'
    obsp: 'connectivities', 'distances'
```

Example 5 (python):
```python
sc.pl.umap(adata,color="fine_grain")
```

Example 6 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 7 (python):
```python
# Add updated label - we have 2 young and 2 old populations

immature_barcodes = annotation_df[annotation_df['AtrialCardiomyocytesImmature_annotation']=="AtrialCardiomyocytesImmature"].index

# Create a dictionary to hold the updates
updates = {}

for barcode in immature_barcodes:
    if barcode in adata.obs['fine_grain'].index:
        fine_grain_annotation = adata.obs['fine_grain'][barcode]
        if fine_grain_annotation == 'AtrialCardiomyocytesLeft':
            updates[barcode] = 'AtrialCardiomyocytesLeft_immature'
        elif fine_grain_annotation == 'AtrialCardiomyocytesRight':
            updates[barcode] = 'AtrialCardiomyocytesRight_immature'

annotation_df['AtrialCardiomyocytesImmature_annotation_v2'] = annotation_df['AtrialCardiomyocytesImmature_annotation']

# Apply the updates from the dictionary
for barcode, new_annotation in updates.items():
    annotation_df.loc[barcode, 'AtrialCardiomyocytesImmature_annotation_v2'] = new_annotation
```

Example 8 (unknown):
```unknown
annotation_df[['AtrialCardiomyocytesImmature_annotation','AtrialCardiomyocytesImmature_annotation_v2']].value_counts()
```

Example 9 (unknown):
```unknown
AtrialCardiomyocytesImmature_annotation  AtrialCardiomyocytesImmature_annotation_v2
AtrialCardiomyocytesLeft                 AtrialCardiomyocytesLeft                      2828
AtrialCardiomyocytesRight                AtrialCardiomyocytesRight                     2509
AtrialCardiomyocytesImmature             AtrialCardiomyocytesLeft_immature              213
                                         AtrialCardiomyocytesRight_immature             114
dtype: int64
```

Example 10 (unknown):
```unknown
annotation_df['AtrialCardiomyocytesImmature_annotation']
```

Example 11 (unknown):
```unknown
barcode
BHF_F_Hea11933667_BHF_F_Hea11596620_CTTCTCAAGCGGTTAT-1       AtrialCardiomyocytesRight
BHF_F_Hea11933674_BHF_F_Hea11596627_GATTAGCTCAGCACGC-1        AtrialCardiomyocytesLeft
BHF_F_Hea11933674_BHF_F_Hea11596627_CATGCAAGTTAGGTGC-1        AtrialCardiomyocytesLeft
BHF_F_Hea11933668_BHF_F_Hea11596621_CCTTACTCAACACCTA-1       AtrialCardiomyocytesRight
BHF_F_Hea11933666_BHF_F_Hea11596619_AGCTTCCTCGTTATAG-1        AtrialCardiomyocytesLeft
                                                                      ...             
BHF_F_Hea11064672_BHF_F_Hea11031825_GCACTTACAGCCAGAA-1        AtrialCardiomyocytesLeft
BHF_F_Hea11933669_BHF_F_Hea11596622_GCTAACAGTTAGAGGG-1       AtrialCardiomyocytesRight
BHF_F_Hea11933673_BHF_F_Hea11596626_GCCTATTGTGTGTCCC-1    AtrialCardiomyocytesImmature
BHF_F_Hea11064671_BHF_F_Hea11031824_CGTCAATAGCTAAGTC-1        AtrialCardiomyocytesLeft
BHF_F_Hea11064671_BHF_F_Hea11031824_CTTACTAGTTGTTGCT-1        AtrialCardiomyocytesLeft
Name: AtrialCardiomyocytesImmature_annotation, Length: 5664, dtype: object
```

Example 12 (python):
```python
# Step 1: Initialize 'fine_grain_modified' with values from 'fine_grain' and convert to string
adata.obs['fine_grain_modified'] = adata.obs['fine_grain'].astype(str)

# Step 2: Update 'fine_grain_modified' for barcodes present in 'annotation_df'
for barcode in annotation_df.index:
    if barcode in adata.obs.index:
        new_value = annotation_df.at[barcode, 'AtrialCardiomyocytesImmature_annotation_v2']
        adata.obs.at[barcode, 'fine_grain_modified'] = new_value

# Optional: Convert 'fine_grain_modified' back to categorical
adata.obs['fine_grain_modified'] = adata.obs['fine_grain_modified'].astype('category')

sc.pl.umap(adata,color="fine_grain_modified")
```

Example 13 (unknown):
```unknown
/lustre/scratch126/cellgen/team205/jc48/miniconda3/envs/omni_env/lib/python3.11/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 14 (python):
```python
adata.obs.fine_grain_modified.to_csv("fine_grain_modified.csv")
```

Example 15 (python):
```python
bdata = adata[adata.obs]
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/AtrialCardiomyocytes/Visium_mapping/Atrial_Proportions_BoxPlot_ratio.html

4100 rows × 8 columns

4100 rows × 9 columns

4100 rows × 15 columns

4100 rows × 15 columns

4100 rows × 15 columns

2523 rows × 15 columns

16400 rows × 3 columns

16400 rows × 3 columns

**Examples:**

Example 1 (python):
```python
import pandas as pd
import plotly.express as px
```

Example 2 (python):
```python
import matplotlib
matplotlib.rcParams['pdf.fonttype'] = 42
matplotlib.rcParams['ps.fonttype'] = 42
```

Example 3 (python):
```python
OCT = pd.read_csv('AtrialProportionsOCT.csv')
OCT['modality']="OCT"
FFPE = pd.read_csv('AtrialProportionsFFPE.csv')
FFPE['modality']="FFPE"
df=pd.concat([OCT,FFPE])
df
```

Example 4 (python):
```python
def age_to_days(age_str):
    # Split the string into parts
    age_type, age_time = age_str.split('-')

    # Initialize weeks and days
    weeks = 0
    days = 0

    # Extract weeks and days from the age_time string
    if 'w' in age_time:
        week_day_parts = age_time.split('w')
        weeks = int(week_day_parts[0])  # Convert weeks to integer
        if len(week_day_parts) > 1 and 'd' in week_day_parts[1]:
            days = int(week_day_parts[1].replace('d', ''))  # Convert days to integer

    # Convert gestational age (GA) to post conception weeks (PCW)
    if age_type == 'GA':
        weeks -= 2

    # Convert weeks and days to total days
    total_days = weeks * 7 + days
    return total_days

# Apply the conversion to the DataFrame
df['days_post_conception'] = df['age'].apply(age_to_days)
df
```

Example 5 (unknown):
```unknown
df.days_post_conception = df.days_post_conception.astype('category')
df.days_post_conception = df.days_post_conception.astype(int)
```

Example 6 (python):
```python
# Adding a new column 'post_conception_weeks' based on the 'days post conception' column
def days_to_weeks_days(days):
    weeks = days // 7
    days_remaining = days % 7
    return f"{weeks}w{days_remaining}d"

df['post_conception_weeks'] = df['days_post_conception'].apply(days_to_weeks_days)

# Displaying the first few rows of the DataFrame to verify the new column
df.head()
```

Example 7 (unknown):
```unknown
# Verifying the changes by inspecting the value counts again
df['post_conception_weeks'].value_counts()
```

Example 8 (unknown):
```unknown
16w0d    2045
7w4d     1210
13w0d     682
5w4d      163
Name: post_conception_weeks, dtype: int64
```

Example 9 (python):
```python
# Correcting the order of the categories for the 'post_conception_weeks' column
correct_order_categories = ['16w0d', '13w0d','7w4d','5w4d']
correct_order_categories = ['5w4d', '7w4d','13w0d','16w0d']
df['post_conception_weeks'] = pd.Categorical(values=df['post_conception_weeks'], categories=correct_order_categories, ordered=True)

# Verifying the changes by inspecting the value counts again
df['post_conception_weeks'].dtypes
```

Example 10 (unknown):
```unknown
CategoricalDtype(categories=['5w4d', '7w4d', '13w0d', '16w0d'], ordered=True)
```

Example 11 (unknown):
```unknown
# Assuming 'data' is your pandas DataFrame
# Calculate the total for Left and Right cells separately
df['AtrialCardiomyocytesLeft_immature'] + df['AtrialCardiomyocytesLeft']
```

Example 12 (unknown):
```unknown
0        2.222819
1        4.122041
2        9.167294
3        1.971105
4        2.066072
          ...    
2040     3.813929
2041     1.510159
2042    16.247391
2043     3.719476
2044    21.171020
Length: 4100, dtype: float64
```

Example 13 (unknown):
```unknown
# Assuming 'data' is your pandas DataFrame
# Calculate the total for Left and Right cells separately
total_left = df['AtrialCardiomyocytesLeft_immature'] + df['AtrialCardiomyocytesLeft']
total_right = df['AtrialCardiomyocytesRight_immature'] + df['AtrialCardiomyocytesRight']

# Calculate the proportions for each cell type relative to their side
df['AtrialCardiomyocytesLeft_immature_prop'] = df['AtrialCardiomyocytesLeft_immature'] / total_left
df['AtrialCardiomyocytesLeft_prop'] = df['AtrialCardiomyocytesLeft'] / total_left

df['AtrialCardiomyocytesRight_immature_prop'] = df['AtrialCardiomyocytesRight_immature'] / total_right
df['AtrialCardiomyocytesRight_prop'] = df['AtrialCardiomyocytesRight'] / total_right

# Check the new columns in the DataFrame
df[['AtrialCardiomyocytesLeft_immature_prop', 'AtrialCardiomyocytesLeft_prop', 'AtrialCardiomyocytesRight_immature_prop', 'AtrialCardiomyocytesRight_prop']]
df
```

Example 14 (unknown):
```unknown
df.AtrialCardiomyocytesLeft_immature.hist(bins=50)
```

Example 15 (python):
```python
import seaborn as sns
import matplotlib.pyplot as plt
```

Example 16 (python):
```python
import pandas as pd

# Specify the columns to check for the condition
columns_to_check = [
    'AtrialCardiomyocytesLeft_immature',
    'AtrialCardiomyocytesLeft',
    'AtrialCardiomyocytesRight',
    'AtrialCardiomyocytesRight_immature'
]

# Add 'filter_out' column based on the condition
df['filter_out'] = df[columns_to_check].apply(lambda x: (x < 0.01).any(), axis=1)
df
```

Example 17 (unknown):
```unknown
df[df['filter_out']==False]
```

Example 18 (unknown):
```unknown
# Assuming 'data' is your DataFrame and it already includes an 'age' column
df_long = df.melt(id_vars='post_conception_weeks', var_name='cell_type', value_name='proportion', 
                    value_vars=['AtrialCardiomyocytesLeft_immature_prop', 'AtrialCardiomyocytesLeft_prop', 
                                'AtrialCardiomyocytesRight_prop', 'AtrialCardiomyocytesRight_immature_prop'])
df_long
```

Example 19 (unknown):
```unknown
#df_long.replace(to_replace='AtrialCardiomyocytesLeft_immature_prop', value='AtrialCardiomyocytesLeft_immature', inplace=True)
df_long.replace(to_replace='AtrialCardiomyocytesLeft_prop', value='AtrialCardiomyocytesLeft_mature_prop', inplace=True)
#df_long.replace(to_replace='AtrialCardiomyocytesRight_immature_prop', value='AtrialCardiomyocytesRight_immature', inplace=True)
df_long.replace(to_replace='AtrialCardiomyocytesRight_prop', value='AtrialCardiomyocytesRight_mature_prop', inplace=True)
df_long
```

Example 20 (python):
```python
# Define the desired order of 'cell_type' labels
cell_type_order = [
    'AtrialCardiomyocytesLeft_immature_prop', 
    'AtrialCardiomyocytesLeft_mature_prop', 
    'AtrialCardiomyocytesRight_immature_prop', 
    'AtrialCardiomyocytesRight_mature_prop'
]

# Convert the 'cell_type' column to a categorical type with the specified order
df_long['cell_type'] = pd.Categorical(df_long['cell_type'], categories=cell_type_order, ordered=True)

# and the legend will be in the second row.
fig, (ax_main, ax_legend) = plt.subplots(2, 1, figsize=(10, 8), 
                                         gridspec_kw={'height_ratios': [4, 1], 'hspace': 0.05})

# The main strip plot without specifying legend=False
sns.stripplot(data=df_long, x='post_conception_weeks', y='proportion', hue='cell_type', dodge=True, ax=ax_main)

# Hide the legend created by seaborn in the main plot
ax_main.get_legend().remove()

# Extract handles and labels for the legend after creating the plot
handles, labels = ax_main.get_legend_handles_labels()

# Creating the legend in the subplot with 2 columns
ax_legend.legend(handles, labels, loc='center', ncol=2, frameon=False)
ax_legend.axis('off')  # Hide the axes of the legend subplot

plt.show()
```

Example 21 (python):
```python
# Creating the figure and axes for the plot and the legend
fig, (ax_main, ax_legend) = plt.subplots(2, 1, figsize=(10, 8), 
                                         gridspec_kw={'height_ratios': [4, 1], 'hspace': 0.05})

# Plotting the box plot first
sns.boxplot(data=df_long, x='post_conception_weeks', y='proportion', hue='cell_type', ax=ax_main, 
            fliersize=0,  # Optionally hide the outliers in the boxplot to avoid overlap with the stripplot points
            dodge=True, saturation = 1)

# Adding the strip plot on top
sns.stripplot(data=df_long, x='post_conception_weeks', y='proportion', hue='cell_type', dodge=True, ax=ax_main,
              alpha=0.3)

# Hide the legend created by seaborn in the main plot to avoid duplication
ax_main.get_legend().remove()

# Extract handles and labels for the legend after creating the plot
handles, labels = ax_main.get_legend_handles_labels()

# Creating the legend in the subplot with 2 columns
ax_legend.legend(handles[:len(cell_type_order)], labels[:len(cell_type_order)], loc='center', ncol=2, frameon=False)
ax_legend.axis('off')  # Hide the axes of the legend subplot
plt.tight_layout()
plt.savefig("./figures/aCMs_immature_proportion_boxplots.pdf",bbox_inches='tight')
plt.show()
```

Example 22 (python):
```python
/tmp/ipykernel_167508/380920426.py:23: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
  plt.tight_layout()
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/no1_compact-vs-trab_gene-signatures.html

**Contents:**
- Function¶
- Read in adata¶
- DEG analysis: compact-vs-trab at the end stage¶
- Read in associated genes¶
- Select associated genes which were highest in the group_a or group_b¶
- Associated genes¶
- Select specific and common ones¶
- Save the genes¶
- Scoring and plot¶

i.e. separating into UP or DOWN regulated genes

**Examples:**

Example 1 (python):
```python
# Parameters
working_dir = "/nfs/team205/heart/Vitro/TF_selection"
path_to_adata_gex_dir = "/nfs/team205/heart/anndata_objects/Vitro/post_multiVI_full-features"
group_key = "t_binned"
group_a = "t3"
group_b = "t1"
time_key = "t"
subset_object = False
geneset_list = ["MSigDB_Hallmark_2020", "KEGG_2021_Human", "GO_Biological_Process_2023"]

path_to_eRegulon_signatures_gene = "/nfs/team205/heart/pyscenic/plus/eRegulon_signatures/vCM_onlyLeft_Gene_eRegulon_signatures.pkl"
path_to_eRegulon_signatures_region = "/nfs/team205/heart/pyscenic/plus/eRegulon_signatures/vCM_onlyLeft_Region_eRegulon_signatures.pkl"
```

Example 2 (python):
```python
import scanpy as sc
import pandas as pd
import numpy as np
import anndata

import matplotlib.pyplot as plt
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 
import seaborn as sns
```

Example 3 (python):
```python
import session_info
session_info.show()
```

Example 4 (python):
```python
-----
anndata             0.9.2
matplotlib          3.7.2
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 5 (python):
```python
PIL                         10.0.0
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.11.17
cffi                        1.15.1
charset_normalizer          3.2.0
colorama                    0.4.6
comm                        0.1.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.8
decorator                   5.1.1
defusedxml                  0.7.1
executing                   1.2.0
fastjsonschema              NA
fqdn                        NA
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_metadata          NA
importlib_resources         NA
ipykernel                   6.25.1
isoduration                 NA
jedi                        0.19.0
jinja2                      3.1.2
joblib                      1.3.2
json5                       NA
jsonpointer                 2.0
jsonschema                  4.19.0
jsonschema_specifications   NA
jupyter_events              0.7.0
jupyter_server              2.7.1
jupyterlab_server           2.24.0
kiwisolver                  1.4.5
leidenalg                   0.10.1
llvmlite                    0.40.1
markupsafe                  2.1.3
mpl_toolkits                NA
natsort                     8.4.0
nbformat                    5.9.2
numba                       0.57.1
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.16.1
pyparsing                   3.0.9
pythonjsonlogger            NA
pytz                        2023.3
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
ruamel                      NA
scipy                       1.10.1
send2trash                  NA
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
sphinxcontrib               NA
stack_data                  0.6.2
statsmodels                 0.14.0
texttable                   1.6.7
threadpoolctl               3.2.0
tornado                     6.3.3
traitlets                   5.9.0
typing_extensions           NA
uri_template                NA
urllib3                     2.0.4
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.2
yaml                        6.0.1
zipp                        NA
zmq                         25.1.1
zstandard                   0.19.0
```

Example 6 (unknown):
```unknown
-----
IPython             8.12.0
jupyter_client      8.3.0
jupyter_core        4.12.0
jupyterlab          4.0.5
notebook            7.0.2
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-4.15.0-220-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-02-13 16:26
```

Example 7 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 8 (python):
```python
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 9 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 10 (python):
```python
path_to_adata_gex_dir
```

Example 11 (unknown):
```unknown
'/nfs/team205/heart/anndata_objects/Vitro/post_multiVI_full-features'
```

Example 12 (python):
```python
adata_compact = sc.read_h5ad(f'{path_to_adata_gex_dir}/Foetal_vCM_Left-compact_rna.h5ad')
# filter and log normalise
# filter
sc.pp.filter_genes(adata_compact, min_cells=3)

# log normalise
sc.pp.normalize_total(adata_compact, target_sum=1e4)
sc.pp.log1p(adata_compact)
print(adata_compact.X.data[:10])

adata_trab = sc.read_h5ad(f'{path_to_adata_gex_dir}/Foetal_vCM_Left-trabeculated_rna.h5ad')
# filter and log normalise
# filter
sc.pp.filter_genes(adata_trab, min_cells=3)

# log normalise
sc.pp.normalize_total(adata_trab, target_sum=1e4)
sc.pp.log1p(adata_trab)
print(adata_trab.X.data[:10])
```

Example 13 (unknown):
```unknown
[1.0619993 1.5653387 1.8985853 1.5653387 2.3476288 1.8985853 1.5653387
 1.0619993 1.0619993 1.0619993]
[1.0934392 1.0934392 1.0934392 1.603227  1.0934392 1.0934392 1.0934392
 1.603227  1.0934392 1.603227 ]
```

Example 14 (python):
```python
sc.pl.draw_graph(adata_compact, color=[group_key, time_key], size=5, wspace=0.3, ncols=3)
```

Example 15 (unknown):
```unknown
/nfs/team205/kk18/miniconda3/envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 16 (python):
```python
sc.pl.draw_graph(adata_trab, color=[group_key, time_key], size=5, wspace=0.3, ncols=3)
```

Example 17 (unknown):
```unknown
/nfs/team205/kk18/miniconda3/envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 18 (python):
```python
# whole data
adata_all = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/vCM/post_scFate/foetal_vCM_onlyLeft.h5ad')

# subset gene expression data
adata_all = adata_all[:,adata_all.var['modality']=='Gene Expression']

# filter and log-normalise
sc.pp.filter_genes(adata_all, min_cells=3)
sc.pp.normalize_total(adata_all, target_sum=1e4)
sc.pp.log1p(adata_all)

print(adata_all.X.data[:10])
adata_all
```

Example 19 (python):
```python
/nfs/team205/kk18/miniconda3/envs/vitro/lib/python3.8/site-packages/scanpy/preprocessing/_simple.py:251: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var['n_cells'] = number
```

Example 20 (unknown):
```unknown
[1.0619993 1.5653387 1.8985853 1.5653387 2.3476288 1.8985853 1.5653387
 1.0619993 1.0619993 1.0619993]
```

Example 21 (unknown):
```unknown
AnnData object with n_obs × n_vars = 16451 × 26187
    obs: 'sangerID', 'combinedID', 'donor', 'region', 'age', 'sex', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'week', 'trimester', 'cycling', 'S_score', 'G2M_score', 'phase', 'fine_grain', 'mid_grain', 'coarse_grain', 'batch_key', 'total_counts', 'total_counts_mt', 'total_counts_ribo', 'n_genes_by_counts', 'pct_counts_mt', 'pct_counts_ribo', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'log1p_total_counts_mt', 'log1p_total_counts_ribo', 'scrublet_score', 'doublet_pval', 'doublet_bh_pval', '_indices', '_scvi_batch', '_scvi_labels', 'leiden_MultiVI', 't', 'seg', 'edge', 'milestones', 'age_days', 'week_cont'
    var: 'modality', 'gene_name_scRNA-0', 'gene_id', 'seqnames', 'start', 'end', 'width', 'strand', 'score', 'replicateScoreQuantile', 'groupScoreQuantile', 'Reproducibility', 'GroupReplicate', 'distToGeneStart', 'nearestGene', 'peakType', 'distToTSS', 'nearestTSS', 'GC', 'idx', 'N', 'n_cells'
    uns: '_scvi_manager_uuid', '_scvi_uuid', 'cycling_colors', 'donor_colors', 'draw_graph', 'fine_grain_colors', 'graph', 'leiden', 'leiden_MultiVI_colors', 'milestones_colors', 'neighbors', 'ppt', 'pseudotime_list', 'region_colors', 'umap', 'week_colors', 'log1p'
    obsm: 'MultiVI_latent', 'X_R', 'X_draw_graph_fa', 'X_palantir', 'X_umap', 'X_umap13', '_scvi_extra_categorical_covs', '_scvi_extra_continuous_covs'
    obsp: 'connectivities', 'distances'
```

Example 22 (python):
```python
adata_all.obs.fine_grain.value_counts()
```

Example 23 (unknown):
```unknown
VentricularCardiomyocytesLeftCompact         11397
VentricularCardiomyocytesLeftTrabeculated     5054
Name: fine_grain, dtype: int64
```

Example 24 (unknown):
```unknown
logFC_threshold = 0.5
```

Example 25 (python):
```python
# make new label for the testing: combining fine_grain and t_binned
## for each compact and trabeculated
adata_compact.obs['fine_grain_time'] = adata_compact.obs['fine_grain'].astype('str')+'_'+adata_compact.obs['t_binned'].astype('str')
adata_trab.obs['fine_grain_time'] = adata_trab.obs['fine_grain'].astype('str')+'_'+adata_trab.obs['t_binned'].astype('str')

## add to the whold anndata
adata_all.obs['fine_grain_time'] = np.nan
adata_all.obs.loc[adata_compact.obs_names,'fine_grain_time'] = adata_compact.obs['fine_grain_time'].copy()
adata_all.obs.loc[adata_trab.obs_names,'fine_grain_time'] = adata_trab.obs['fine_grain_time'].copy()

# checking all cells has a label
all(adata_all.obs['fine_grain_time'].isna()==False)
```

Example 26 (python):
```python
# compact, up
compact_deg_up = utils.rank_genes_groups_with_mean(adata_all,
                                  groupby='fine_grain_time',
                                  groups=['VentricularCardiomyocytesLeftCompact_t3'],
                                  reference='VentricularCardiomyocytesLeftTrabeculated_t3',
                                  method='t-test',
                                  log2fc_min=logFC_threshold ## thresholdng for UP
                                 )
compact_deg_up = compact_deg_up.sort_values('scores',ascending=False)
compact_deg_up[:20]
```

Example 27 (python):
```python
# compact, down
compact_deg_down = utils.rank_genes_groups_with_mean(adata_all,
                                  groupby='fine_grain_time',
                                  groups=['VentricularCardiomyocytesLeftCompact_t3'],
                                  reference='VentricularCardiomyocytesLeftTrabeculated_t3',
                                  method='t-test',
                                  log2fc_max = -logFC_threshold ## thresholdng for DOWN
                                 )
compact_deg_down = compact_deg_down.sort_values('scores',ascending=True)
compact_deg_down[:20]
```

Example 28 (python):
```python
# trabeculated, up
trab_deg_up = utils.rank_genes_groups_with_mean(adata_all,
                                  groupby='fine_grain_time',
                                  groups=['VentricularCardiomyocytesLeftTrabeculated_t3'],
                                  reference='VentricularCardiomyocytesLeftCompact_t3',
                                  method='t-test',
                                  log2fc_min=logFC_threshold ## thresholdng for UP
                                 )
trab_deg_up = trab_deg_up.sort_values('scores',ascending=False)
trab_deg_up[:20]
```

Example 29 (python):
```python
# trabeculated, down
trab_deg_down = utils.rank_genes_groups_with_mean(adata_all,
                                  groupby='fine_grain_time',
                                  groups=['VentricularCardiomyocytesLeftTrabeculated_t3'],
                                  reference='VentricularCardiomyocytesLeftCompact_t3',
                                  method='t-test',
                                  log2fc_max = -logFC_threshold ## thresholdng for DOWN
                                 )
trab_deg_down = trab_deg_down.sort_values('scores',ascending=True)
trab_deg_down[:20]
```

Example 30 (python):
```python
fit_res_compact = pd.read_csv(f'{working_dir}/foetal_vCM_left-compact/outs/Fitting_{time_key}_Gene.csv',index_col=0)
fit_res_trab = pd.read_csv(f'{working_dir}/foetal_vCM_left-trabeculated/outs/Fitting_{time_key}_Gene.csv',index_col=0)
```

Example 31 (unknown):
```unknown
fit_res_compact.loc['SLC27A1']
```

Example 32 (unknown):
```unknown
p_val                       0.0
A                      0.350825
fdr                         0.0
st                            0
signi                     False
activator_repressor     SLC27A1
Name: SLC27A1, dtype: object
```

Example 33 (unknown):
```unknown
fit_res_trab.loc['SLC27A1']
```

Example 34 (unknown):
```unknown
p_val                   0.00005
A                      0.967977
fdr                         1.0
st                            0
signi                     False
activator_repressor     SLC27A1
Name: SLC27A1, dtype: object
```

Example 35 (python):
```python
# compact
group_mean_compact = utils.grouped_obs_mean(adata_compact,group_key=group_key)

# select upregulated (a>b) or downregulated (b>a) genes
group_a_genes_compact = list(group_mean_compact.index[group_mean_compact[group_a] > group_mean_compact[group_b]])
group_b_genes_compact = list(group_mean_compact.index[group_mean_compact[group_b] > group_mean_compact[group_a]])

group_mean_compact.head()
```

Example 36 (unknown):
```unknown
group_mean_compact.loc['SLC27A1']
```

Example 37 (unknown):
```unknown
t1    0.054584
t2    0.130589
t3    0.216747
Name: SLC27A1, dtype: float64
```

Example 38 (python):
```python
# trabeculated
group_mean_trab = utils.grouped_obs_mean(adata_trab,group_key=group_key)

# select upregulated (a>b) or downregulated (b>a) genes
group_a_genes_trab = list(group_mean_trab.index[group_mean_trab[group_a] > group_mean_trab[group_b]])
group_b_genes_trab = list(group_mean_trab.index[group_mean_trab[group_b] > group_mean_trab[group_a]])

group_mean_trab.head()
```

Example 39 (unknown):
```unknown
group_mean_trab.loc['SLC27A1']
```

Example 40 (unknown):
```unknown
t1    0.025651
t2    0.154269
t3    0.204383
Name: SLC27A1, dtype: float64
```

Example 41 (unknown):
```unknown
fit_fdr_threshold = 0.05
fit_A_threshold = 0.5
```

Example 42 (python):
```python
# compact
fit_res_sel_compact = fit_res_compact[(fit_res_compact['fdr']<fit_fdr_threshold)&\
                                      (fit_res_compact['A']>fit_A_threshold)
                                     ]
# group a, upregulated
shared = list(set(fit_res_sel_compact.index).intersection(group_a_genes_compact))
fit_res_sel_up2a_compact = fit_res_sel_compact.loc[shared]
fit_res_sel_up2a_compact = fit_res_sel_up2a_compact.sort_values('A',ascending=False)
print(f'up towards {group_a}, number of fitted genes: {len(fit_res_sel_up2a_compact)}')
# group b, downregulated
shared = list(set(fit_res_sel_compact.index).intersection(group_b_genes_compact))
fit_res_sel_up2b_compact = fit_res_sel_compact.loc[shared]
fit_res_sel_up2b_compact = fit_res_sel_up2b_compact.sort_values('A',ascending=False)
print(f'up towards {group_b}, number of fitted genes: {len(fit_res_sel_up2b_compact)}')
```

Example 43 (unknown):
```unknown
up towards t3, number of fitted genes: 1073
up towards t1, number of fitted genes: 2176
```

Example 44 (python):
```python
# trabeculated
fit_res_sel_trab = fit_res_trab[(fit_res_trab['fdr']<fit_fdr_threshold)&\
                                (fit_res_trab['A']>fit_A_threshold)
                                     ]
# group a, upregulated
shared = list(set(fit_res_sel_trab.index).intersection(group_a_genes_trab))
fit_res_sel_up2a_trab = fit_res_sel_trab.loc[shared]
fit_res_sel_up2a_trab = fit_res_sel_up2a_trab.sort_values('A',ascending=False)
print(f'up towards {group_a}, number of fitted genes: {len(fit_res_sel_up2a_trab)}')
# group b, downregulated
shared = list(set(fit_res_sel_trab.index).intersection(group_b_genes_trab))
fit_res_sel_up2b_trab = fit_res_sel_trab.loc[shared]
fit_res_sel_up2b_trab = fit_res_sel_up2b_trab.sort_values('A',ascending=False)
print(f'up towards {group_b}, number of fitted genes: {len(fit_res_sel_up2b_trab)}')
```

Example 45 (unknown):
```unknown
up towards t3, number of fitted genes: 893
up towards t1, number of fitted genes: 1972
```

Example 46 (unknown):
```unknown
fit_res_sel_up2a_compact.loc['PRDM16']
```

Example 47 (unknown):
```unknown
p_val                       0.0
A                      1.618132
fdr                         0.0
st                            1
signi                      True
activator_repressor      PRDM16
Name: PRDM16, dtype: object
```

Example 48 (unknown):
```unknown
fit_res_sel_up2a_trab.loc['PRDM16']
```

Example 49 (unknown):
```unknown
p_val                      0.0
A                      1.63836
fdr                        0.0
st                           1
signi                     True
activator_repressor     PRDM16
Name: PRDM16, dtype: object
```

Example 50 (python):
```python
# up toward group a, up-regulated
compact_spec_up2a_genes = list(set(fit_res_sel_up2a_compact.index).intersection(list(compact_deg_up['names'])))
trab_spec_up2a_genes = list(set(fit_res_sel_up2a_trab.index).intersection(list(trab_deg_up['names'])))
common_up2a_genes = list(set(fit_res_sel_up2a_compact.index).intersection(fit_res_sel_up2a_trab.index))
common_up2a_genes = list(set(common_up2a_genes)-set(compact_deg_up['names'])-set(trab_deg_up['names'])) ### removing DEGs
print(f'up toward {group_a}')
print(len(compact_spec_up2a_genes))
print(len(trab_spec_up2a_genes))
print(len(common_up2a_genes))

# up toward group b, down-regulated
compact_spec_up2b_genes = list(set(fit_res_sel_up2b_compact.index).intersection(list(compact_deg_down['names'])))
trab_spec_up2b_genes = list(set(fit_res_sel_up2b_trab.index).intersection(list(trab_deg_down['names'])))
common_up2b_genes = list(set(fit_res_sel_up2b_compact.index).intersection(fit_res_sel_up2b_trab.index))
common_up2b_genes = list(set(common_up2b_genes)-set(compact_deg_down['names'])-set(trab_deg_down['names'])) ### removing DEGs
print(f'up toward {group_b}')
print(len(compact_spec_up2b_genes))
print(len(trab_spec_up2b_genes))
print(len(common_up2b_genes))
```

Example 51 (unknown):
```unknown
up toward t3
130
107
336
up toward t1
185
384
738
```

Example 52 (python):
```python
# venn diagram for upregulated genes
# just checking there is no overlap
from matplotlib_venn import venn3
venn3((set(compact_spec_up2a_genes),set(trab_spec_up2a_genes),set(common_up2a_genes)),
      set_labels=('compact', 'trab', 'common')
     )
```

Example 53 (python):
```python
<matplotlib_venn._common.VennDiagram at 0x14cfcef75a60>
```

Example 54 (python):
```python
# venn diagram for downregulated genes
# just checking there is no overlap
from matplotlib_venn import venn3
venn3((set(compact_spec_up2b_genes),set(trab_spec_up2b_genes),set(common_up2b_genes)),
      set_labels=('compact', 'trab', 'common')
     )
```

Example 55 (python):
```python
<matplotlib_venn._common.VennDiagram at 0x14cfcf01eeb0>
```

Example 56 (python):
```python
gene_signatures = {}
# up, towards group_a
gene_signatures['compact_specific_up'] = compact_spec_up2a_genes
gene_signatures['trab_specific_up'] = trab_spec_up2a_genes
gene_signatures['common_up'] = common_up2a_genes

# down, towards group_b
gene_signatures['compact_specific_down'] = compact_spec_up2b_genes
gene_signatures['trab_specific_down'] = trab_spec_up2b_genes
gene_signatures['common_down'] = common_up2b_genes

print(gene_signatures.keys())

# save
utils.save_pkl(gene_signatures,
           file = f'{working_dir}/foetal_vCM_left_compact-vs-trabeculated_downstream/gene_signatures.pkl'
          )
```

Example 57 (unknown):
```unknown
dict_keys(['compact_specific_up', 'trab_specific_up', 'common_up', 'compact_specific_down', 'trab_specific_down', 'common_down'])
```

Example 58 (unknown):
```unknown
f'{working_dir}/foetal_vCM_left_compact-vs-trabeculated_downstream/gene_signatures.pkl'
```

Example 59 (unknown):
```unknown
'/nfs/team205/heart/Vitro/TF_selection/foetal_vCM_left_compact-vs-trabeculated_downstream/gene_signatures.pkl'
```

Example 60 (python):
```python
adata_all.X.data[:10]
```

Example 61 (unknown):
```unknown
array([1.0619993, 1.5653387, 1.8985853, 1.5653387, 2.3476288, 1.8985853,
       1.5653387, 1.0619993, 1.0619993, 1.0619993], dtype=float32)
```

Example 62 (python):
```python
# upregulated
sc.tl.score_genes(adata_all,compact_spec_up2a_genes,score_name=f'compact ({len(compact_spec_up2a_genes)}g)')
sc.tl.score_genes(adata_all,trab_spec_up2a_genes,score_name=f'trabeculated ({len(trab_spec_up2a_genes)}g)')
sc.tl.score_genes(adata_all,common_up2a_genes,score_name=f'common ({len(common_up2a_genes)}g)')

sc.pl.draw_graph(adata_all, color=['fine_grain',
                                   f'compact ({len(compact_spec_up2a_genes)}g)',
                                   f'trabeculated ({len(trab_spec_up2a_genes)}g)',
                                   f'common ({len(common_up2a_genes)}g)',
                                  ], 
                 frameon=False, legend_loc='on data',legend_fontsize=0,
                 size=7, wspace=0.1, ncols=4, cmap='YlOrRd',show=False)
plt.savefig(f"{figdir}/vCMs-left_force-directed_gene-signatures-upregulated.pdf", bbox_inches="tight",dpi=300)
```

Example 63 (unknown):
```unknown
/nfs/team205/kk18/miniconda3/envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

Example 64 (python):
```python
# downregulated
sc.tl.score_genes(adata_all,compact_spec_up2b_genes,score_name=f'compact ({len(compact_spec_up2b_genes)}g)')
sc.tl.score_genes(adata_all,trab_spec_up2b_genes,score_name=f'trabeculated ({len(trab_spec_up2b_genes)}g)')
sc.tl.score_genes(adata_all,common_up2b_genes,score_name=f'common ({len(common_up2b_genes)}g)')

sc.pl.draw_graph(adata_all, color=['fine_grain',
                                   f'compact ({len(compact_spec_up2b_genes)}g)',
                                   f'trabeculated ({len(trab_spec_up2b_genes)}g)',
                                   f'common ({len(common_up2b_genes)}g)',
                                  ], 
                 frameon=False, legend_loc='on data',legend_fontsize=0,
                 size=7, wspace=0.1, ncols=4, cmap='YlOrRd',show=False)
plt.savefig(f"{figdir}/vCMs-left_force-directed_gene-signatures-downregulated.pdf", bbox_inches="tight",dpi=300)
```

Example 65 (unknown):
```unknown
/nfs/team205/kk18/miniconda3/envs/vitro/lib/python3.8/site-packages/scanpy/plotting/_tools/scatterplots.py:392: UserWarning: No data for colormapping provided via 'c'. Parameters 'cmap' will be ignored
  cax = scatter(
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig4_Cardiomyocytes/VentricularCardiomyocytes/TransmuralAxis/tm_axis_gene_cell-cycle-genes.html

**Contents:**
- Import modules¶
- Functions¶
- Read in adata¶
- Add axis¶
- Differential test¶
- Slect genes to plot¶
- Pathway analysis of outer_up genes¶
- Get genes involved in cell proliferation pathway¶
- Manually curated cardiomyocyte cell cycle genes¶
- Plot genes¶

**Examples:**

Example 1 (python):
```python
import os
import numpy as np
import scanpy as sc
import anndata
import pandas as pd
import matplotlib as mpl
# enables correct plotting of text
mpl.rcParams['pdf.fonttype'] = 42
mpl.rcParams['ps.fonttype'] = 42 
import matplotlib.pyplot as plt
import seaborn as sns

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter("ignore", UserWarning)
```

Example 2 (python):
```python
import session_info
session_info.show()
```

Example 3 (python):
```python
-----
anndata             0.9.2
matplotlib          3.7.2
numpy               1.24.4
pandas              1.5.3
scanpy              1.9.3
seaborn             0.12.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                         10.0.0
anyio                       NA
arrow                       1.2.3
asttokens                   NA
attr                        23.1.0
attrs                       23.1.0
babel                       2.12.1
backcall                    0.2.0
brotli                      1.0.9
certifi                     2023.11.17
cffi                        1.15.1
charset_normalizer          3.2.0
colorama                    0.4.6
comm                        0.1.4
cycler                      0.10.0
cython_runtime              NA
dateutil                    2.8.2
debugpy                     1.6.8
decorator                   5.1.1
defusedxml                  0.7.1
executing                   1.2.0
fastjsonschema              NA
fqdn                        NA
h5py                        3.9.0
idna                        3.4
igraph                      0.10.6
importlib_metadata          NA
importlib_resources         NA
ipykernel                   6.25.1
isoduration                 NA
jedi                        0.19.0
jinja2                      3.1.2
joblib                      1.3.2
json5                       NA
jsonpointer                 2.0
jsonschema                  4.19.0
jsonschema_specifications   NA
jupyter_events              0.7.0
jupyter_server              2.7.1
jupyterlab_server           2.24.0
kiwisolver                  1.4.5
leidenalg                   0.10.1
llvmlite                    0.40.1
markupsafe                  2.1.3
mpl_toolkits                NA
natsort                     8.4.0
nbformat                    5.9.2
numba                       0.57.1
overrides                   NA
packaging                   23.1
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
plotly                      5.16.1
prometheus_client           NA
prompt_toolkit              3.0.39
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.16.1
pyparsing                   3.0.9
pythonjsonlogger            NA
pytz                        2023.3
referencing                 NA
requests                    2.31.0
rfc3339_validator           0.1.4
rfc3986_validator           0.1.1
rpds                        NA
ruamel                      NA
scipy                       1.10.1
send2trash                  NA
six                         1.16.0
sklearn                     1.3.0
sniffio                     1.3.0
socks                       1.7.1
sphinxcontrib               NA
stack_data                  0.6.2
statsmodels                 0.14.0
texttable                   1.6.7
threadpoolctl               3.2.0
tornado                     6.3.3
traitlets                   5.9.0
typing_extensions           NA
uri_template                NA
urllib3                     2.0.4
wcwidth                     0.2.6
webcolors                   1.13
websocket                   1.6.2
yaml                        6.0.1
zipp                        NA
zmq                         25.1.1
zstandard                   0.19.0
```

Example 5 (unknown):
```unknown
-----
IPython             8.12.0
jupyter_client      8.3.0
jupyter_core        4.12.0
jupyterlab          4.0.5
notebook            7.0.2
-----
Python 3.8.17 (default, Jul  5 2023, 21:04:15) [GCC 11.2.0]
Linux-5.4.0-171-generic-x86_64-with-glibc2.17
-----
Session information updated at 2024-03-21 10:38
```

Example 6 (python):
```python
import importlib.util
import sys

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)

spec = importlib.util.spec_from_file_location("module.name", "/nfs/team205/kk18/function/python/tissueaxis.py")
tissueaxis = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = tissueaxis
spec.loader.exec_module(tissueaxis)
```

Example 7 (python):
```python
sc.settings.set_figure_params(dpi=80,dpi_save=300)
```

Example 8 (unknown):
```unknown
figdir = '/nfs/team205/heart/fetal_ms_figures/raw_plots_kk'
```

Example 9 (python):
```python
# This function selects the data for multiple slides from the spatial anndata object.
# Modified `select_slide` in cell2loc for multiple slides

# @param adata: anndata object with multiple spatial experiments
# @param s: list of selected slides
# @param s_col: column in adata.obs listing experiment name for each location

def select_multislide(adata, s, s_col='sample'):

    slides = adata[adata.obs[s_col].isin(s), :]
    s_keys = list(slides.uns['spatial'].keys())
    
    uns_dict = {}
    for ss in s:
        s_spatial=np.array(s_keys)[[ss in k for k in s_keys]][0]
        uns_dict[s_spatial]=slides.uns['spatial'][s_spatial]
    
    slides.uns['spatial'] = uns_dict

    return slides
```

Example 10 (python):
```python
adata_oct_healthy = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT_foetal_raw.h5ad')
print(adata_oct_healthy.shape)
adata_oct_healthy_LV = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/visium-OCT-LV_foetal_raw.h5ad')
print(adata_oct_healthy_LV.shape)
```

Example 11 (unknown):
```unknown
(14261, 33538)
(6767, 33538)
```

Example 12 (python):
```python
# concatenate oct adata
adata_oct = adata_oct_healthy.concatenate(
    [adata_oct_healthy_LV],
    batch_key=None,
    index_unique=None,
    join="inner"
)
```

Example 13 (python):
```python
# log-normalise
sc.pp.normalize_total(adata_oct, target_sum=1e4)
sc.pp.log1p(adata_oct)

# sc.pp.normalize_total(adata_ffpe, target_sum=1e4)
# sc.pp.log1p(adata_ffpe)

# reset var_names
adata_oct.var.set_index('SYMBOL',inplace=True)
# adata_ffpe.var.set_index('SYMBOL',inplace=True)
```

Example 14 (unknown):
```unknown
!ls /nfs/team205/heart/visium_axis/foetal/transmural_LV
```

Example 15 (unknown):
```unknown
BHF_F_Hea13295830.csv	HCAHeartST10317186.csv	HCAHeartST13162330.csv
BHF_F_Hea13295831.csv	HCAHeartST11605165.csv	HCAHeartST13162331.csv
HCAHeartST10317184.csv	HCAHeartST11605167.csv	HCAHeartST13162336.csv
HCAHeartST10317185.csv	HCAHeartST11605168.csv
```

Example 16 (python):
```python
# add
oct_samples = list(set(adata_oct.obs['sample']))

adata_oct.obs['transmural_axis'] = np.nan

axis_files = os.listdir('/nfs/team205/heart/visium_axis/foetal/transmural_LV')
for file in axis_files:
    # read in
    axis = pd.read_csv(f'/nfs/team205/heart/visium_axis/foetal/transmural_LV/{file}',index_col=0)
    # add to anndata
    if file.split('.csv')[0] in oct_samples:
        adata_oct.obs.loc[axis.index,'transmural_axis'] = axis.copy()

# select spots with axis
adata_oct = adata_oct[adata_oct.obs['transmural_axis'].isna()==False]
```

Example 17 (python):
```python
# make bin
adata_oct.obs['transmural_axis_bin'] = pd.cut(adata_oct.obs['transmural_axis'],bins=15)
```

Example 18 (python):
```python
# annotate bin, OCT
adata_oct.obs['tm_bin_annotations'] = adata_oct.obs['transmural_axis_bin'].astype('str').copy()
adata_oct.obs.replace({'tm_bin_annotations':{
    '(-1.002, -0.867]':'outer',
    '(-0.867, -0.733]':'outer',
    '(-0.733, -0.6]':'outer',
    '(-0.6, -0.467]':'outer',
    '(-0.467, -0.333]':'outer',
    '(-0.333, -0.2]':'middle',
    '(-0.2, -0.0667]':'middle',
    '(-0.0667, 0.0667]':'middle',
    '(0.0667, 0.2]':'middle',
    '(0.2, 0.333]':'middle',
    '(0.333, 0.467]':'inner',
    '(0.467, 0.6]':'inner',
    '(0.6, 0.733]':'inner',
    '(0.733, 0.867]':'inner',
    '(0.867, 1.0]':'inner',
}}, inplace=True)

set(adata_oct.obs['tm_bin_annotations'])
```

Example 19 (unknown):
```unknown
{'inner', 'middle', 'outer'}
```

Example 20 (python):
```python
# mean value per group
group_mean_val = utils.grouped_obs_mean(adata_oct, group_key='tm_bin_annotations')
# DE test
sc.tl.rank_genes_groups(adata_oct, 'tm_bin_annotations', method='t-test')
res = sc.get.rank_genes_groups_df(adata_oct, group=None, pval_cutoff=0.05)
# add mean gene expression
df = group_mean_val.reset_index().melt(id_vars=['SYMBOL'])
df.rename(columns={'SYMBOL':'names','variable':'group','value':'mean_gene_exp'},
          inplace=True)
diff_res = res.merge(df, how='left', on=['group','names'])
diff_res.head()
```

Example 21 (python):
```python
sns.scatterplot(data=diff_res, x='logfoldchanges',y='mean_gene_exp')
```

Example 22 (unknown):
```unknown
<Axes: xlabel='logfoldchanges', ylabel='mean_gene_exp'>
```

Example 23 (python):
```python
sns.scatterplot(data=diff_res, x='scores',y='mean_gene_exp')
```

Example 24 (unknown):
```unknown
<Axes: xlabel='scores', ylabel='mean_gene_exp'>
```

Example 25 (unknown):
```unknown
display(diff_res[diff_res['names']=='NRG1'])
```

Example 26 (unknown):
```unknown
['outer', 'middle', 'inner']
```

Example 27 (python):
```python
groups = ['outer','middle','inner']
logfc_thresh = 0.1
mean_exp_thresh = 0.05

gene_lists = {}
for grp in groups:
    gene_lists[f'{grp}_up'] = np.nan
    gene_lists[f'{grp}_down'] = np.nan

for grp in groups:
    # up genes
    genes = diff_res[(diff_res['group']==grp)&\
                (diff_res['logfoldchanges'] > logfc_thresh)&\
                (diff_res['mean_gene_exp'] > mean_exp_thresh)
               ]['names']
    gene_lists[f'{grp}_up'] = list(genes)
    print(f"{grp}_up: {len(gene_lists[f'{grp}_up'])}")

    # down genes
    genes = diff_res[(diff_res['group']==grp)&\
                (diff_res['logfoldchanges'] < -logfc_thresh)
                # (res['mean_gene_exp'] > mean_exp_thresh)
               ]['names']
    gene_lists[f'{grp}_down'] = list(genes)
    print(f"{grp}_down: {len(gene_lists[f'{grp}_down'])}")
```

Example 28 (unknown):
```unknown
outer_up: 837
outer_down: 2790
middle_up: 151
middle_down: 373
inner_up: 2447
inner_down: 1187
```

Example 29 (unknown):
```unknown
gene_lists.keys()
```

Example 30 (unknown):
```unknown
dict_keys(['outer_up', 'outer_down', 'middle_up', 'middle_down', 'inner_up', 'inner_down'])
```

Example 31 (python):
```python
import gseapy as gp
pd.set_option('display.max_rows', None)

geneset_list = ['MSigDB_Hallmark_2020','KEGG_2021_Human','GO_Biological_Process_2023']

enr_res = {}
key = 'outer_up'
geneset = 'GO_Biological_Process_2023'

enr_res = gp.enrichr(gene_list=gene_lists[key],
                 gene_sets=geneset_list,
                 organism='human', # don't forget to set organism to the one you desired! e.g. Yeast
                 outdir=None, # don't write to disk
                )

enr_res_sel = enr_res.results[enr_res.results['Gene_set'] == geneset]
enr_res_sel = enr_res_sel[enr_res_sel['Adjusted P-value'] < 0.01]
enr_res_sel['n_genes'] = [int(x.split('/')[0]) for x in df['Overlap']]
display(enr_res_sel)
```

Example 32 (unknown):
```unknown
'Positive Regulation Of Leukocyte'.lower()
```

Example 33 (unknown):
```unknown
'positive regulation of leukocyte'
```

Example 34 (python):
```python
pd.set_option('max_colwidth', -1)
keys_to_grep = ['cell cycle','proliferation']
mask = [any([y.lower() in x.lower() for y in keys_to_grep]) for x in enr_res_sel['Term']]
enr_res_sel[mask][['Term','n_genes']]
```

Example 35 (python):
```python
terms_of_interest = ['Positive Regulation Of Cell Population Proliferation (GO:0008284)','Regulation Of Cell Population Proliferation (GO:0042127)']
l = list(enr_res_sel[enr_res_sel['Term'].isin(terms_of_interest)]['Genes'])
genes_of_interest = []
for g in l:
    genes_of_interest = genes_of_interest + g.split(';')
genes_of_interest = list(set(genes_of_interest))
print(len(genes_of_interest))
genes_of_interest
```

Example 36 (unknown):
```unknown
['VIPR1',
 'PHIP',
 'PPP2R5C',
 'STRN',
 'CDK6',
 'RPS4X',
 'ZNF503',
 'PIM1',
 'LIFR',
 'CDCA7L',
 'CSF1R',
 'PDPN',
 'THBS1',
 'SQLE',
 'EGLN3',
 'PDGFD',
 'HMGB2',
 'IRF6',
 'NAMPT',
 'PDCD10',
 'PTN',
 'CTBP2',
 'JUN',
 'AIF1',
 'ERBB4',
 'CDKN1A',
 'ZNF268',
 'JAK2',
 'RAPGEF2',
 'ITGB1BP1',
 'NAP1L1',
 'PDGFRB',
 'SGK1',
 'ADGRG1',
 'PDGFC',
 'CD74',
 'ZFP36L1',
 'CD200',
 'FOLR2',
 'EMILIN2',
 'MDK',
 'FGF2',
 'CCNA2',
 'PTPRK',
 'PDGFRA',
 'VEGFC',
 'WT1',
 'KDR',
 'SH3BP4',
 'KDM1A',
 'TDGF1',
 'PDGFB',
 'RPS6',
 'HMGA2',
 'RPS15A',
 'WNT5A',
 'FGF18',
 'SMARCC1',
 'NDUFS4',
 'HSPA1B',
 'CLDN5',
 'CTSH',
 'VEGFA',
 'NR4A3',
 'SFRP1',
 'RPL17',
 'KITLG',
 'HDAC2',
 'PODN',
 'HILPDA',
 'KDM4C',
 'NCK2',
 'SDCBP',
 'ALDH1A2',
 'RGCC',
 'NPM1']
```

Example 37 (python):
```python
cc_genes_df = pd.read_csv('./Gene_markers_CellCycle.csv')
cc_genes = list(cc_genes_df[cc_genes_df['Reference related to CM'].isna()==False]['Gene'])
cc_genes
```

Example 38 (unknown):
```unknown
['CDKN1A',
 'CDKN2A',
 'CDKN2B',
 'CDKN1B',
 'CDKN1C',
 'CDK1',
 'CDK2',
 'CDK4',
 'CDK6',
 'CDK7',
 'CCNA2',
 'CCNB1',
 'CCND1',
 'CCND2',
 'CCNE1',
 'CCNH',
 'YAP1',
 'TBX20']
```

Example 39 (python):
```python
# cc_genes which are statistically enriched
# up in outer side
# or down in inner side

groups = ['outer','middle','inner']
logfc_thresh = 0
mean_exp_thresh = 0

gene_lists = {}
for grp in groups:
    gene_lists[f'{grp}_up'] = np.nan
    gene_lists[f'{grp}_down'] = np.nan

for grp in groups:
    # up genes
    genes = diff_res[(diff_res['group']==grp)&\
                (diff_res['logfoldchanges'] > logfc_thresh)&\
                (diff_res['mean_gene_exp'] > mean_exp_thresh)
               ]['names']
    gene_lists[f'{grp}_up'] = list(genes)
    print(f"{grp}_up: {len(gene_lists[f'{grp}_up'])}")

    # down genes
    genes = diff_res[(diff_res['group']==grp)&\
                (diff_res['logfoldchanges'] < -logfc_thresh)
                # (res['mean_gene_exp'] > mean_exp_thresh)
               ]['names']
    gene_lists[f'{grp}_down'] = list(genes)
    print(f"{grp}_down: {len(gene_lists[f'{grp}_down'])}")

set(cc_genes).intersection(gene_lists['outer_up']+gene_lists['inner_down'])
```

Example 40 (unknown):
```unknown
outer_up: 1058
outer_down: 2798
middle_up: 157
middle_down: 376
inner_up: 2757
inner_down: 1207
```

Example 41 (unknown):
```unknown
{'CCNA2', 'CDK1', 'CDK2', 'CDK6', 'CDKN1A', 'CDKN1C'}
```

Example 42 (unknown):
```unknown
diff_res[diff_res['names']=='CDK2']
```

Example 43 (python):
```python
tissueaxis.plot_axis(adata_oct,
              axis_col='transmural_axis',
              n_bin=15,
              feature_type='gene', # 'gene' or 'celltype'
              features_to_plot=cc_genes,
              features_to_remove=None,
              threshold_express_prop=None, # at quantile=0.9
              threshold_cell_abundance=None, # at quantile=0.9
              correlation=False,
              min_max_scale=True,
              row_cluster=False,
              figsize=(4,5),
              cbar_pos=(-0.2, 0.5, 0.05, 0.18),
              cmap='viridis',
          xticklabels=False,
          yticklabels=True,
              title=None,
              return_feature = False,
                     save=f'{figdir}/tm-axis_OCT_genes_CM-cell-cycle.pdf',
    save_dpi=300,
             )
```

---
