# Developing-Heart-Local-Improved - Rna Qc

**Pages:** 15

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/QC/Foetal_Multiome-RNA_No2_QC-filter.html

**Contents:**
- Import modules¶
- Functions¶
- Read in anndata¶
- Remove doublets¶
- Remove low quality cells¶
- Compute QC stats¶
- Remove cells exceeding the ribo/mito threshold¶
- Calculate RBC score¶
- Remove RBCs¶
- Remove donor-unassigned nuclei in multiplexed samples¶
- Update metadata for the multiplexed samples¶
- Save post-filtered anndata¶
- Just checking¶

**Examples:**

Example 1 (python):
```python
import numpy as np
import pandas as pd
import scanpy as sc
import matplotlib.pyplot as plt
import scrublet as scr
import session_info
```

Example 2 (python):
```python
sc.settings.verbosity = 3
sc.settings.set_figure_params(dpi = 160, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.3.4
numpy               1.20.1
pandas              1.1.4
scanpy              1.7.1
scrublet            NA
session_info        1.0.0
-----
```

Example 4 (unknown):
```unknown
PIL                 8.1.2
anndata             0.7.5
anyio               NA
attr                20.3.0
babel               2.9.0
backcall            0.2.0
brotli              NA
cairo               1.20.0
certifi             2020.12.05
cffi                1.14.5
chardet             4.0.0
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.0
dask                2021.03.1
dateutil            2.8.1
decorator           4.4.2
fsspec              0.8.7
get_version         2.1
google              NA
h5py                2.10.0
idna                2.10
igraph              0.8.3
ipykernel           5.5.0
ipython_genutils    0.2.0
jedi                0.18.0
jinja2              2.11.3
joblib              1.0.1
json5               NA
jsonschema          3.2.0
jupyter_server      1.4.1
jupyterlab_server   2.3.0
kiwisolver          1.3.1
legacy_api_wrap     0.0.0
leidenalg           0.8.3
llvmlite            0.34.0
louvain             0.7.0
markupsafe          1.1.1
mpl_toolkits        NA
natsort             7.1.1
nbclassic           NA
nbformat            5.1.2
numba               0.51.2
numexpr             2.7.3
packaging           20.9
parso               0.8.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
prometheus_client   NA
prompt_toolkit      3.0.16
psutil              5.8.0
ptyprocess          0.7.0
pvectorc            NA
pyarrow             0.16.0
pygments            2.8.0
pyparsing           2.4.7
pyrsistent          NA
pytoml              NA
pytz                2021.1
requests            2.25.1
ruamel              NA
scipy               1.6.1
send2trash          NA
setuptools_scm      NA
simplejson          3.17.2
sinfo               0.3.1
six                 1.15.0
sklearn             0.22
sniffio             1.2.0
socks               1.7.1
sphinxcontrib       NA
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
tlz                 0.11.0
toolz               0.11.1
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
uritemplate         3.0.1
urllib3             1.26.3
wcwidth             0.2.5
yaml                5.1.2
zmq                 22.0.3
```

Example 5 (unknown):
```unknown
-----
IPython             7.21.0
jupyter_client      6.1.11
jupyter_core        4.7.1
jupyterlab          3.0.9
notebook            6.2.0
-----
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0]
Linux-4.15.0-112-generic-x86_64-with-glibc2.10
-----
Session information updated at 2023-02-08 22:44
```

Example 6 (python):
```python
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/QC/multiome_RNA_foetal_prefilter.h5ad')
adata
```

Example 7 (unknown):
```unknown
AnnData object with n_obs × n_vars = 721979 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-10', 'ambient_expression-11', 'ambient_expression-12', 'ambient_expression-13', 'ambient_expression-14', 'ambient_expression-15', 'ambient_expression-16', 'ambient_expression-17', 'ambient_expression-18', 'ambient_expression-19', 'ambient_expression-2', 'ambient_expression-20', 'ambient_expression-21', 'ambient_expression-22', 'ambient_expression-23', 'ambient_expression-24', 'ambient_expression-25', 'ambient_expression-26', 'ambient_expression-27', 'ambient_expression-28', 'ambient_expression-29', 'ambient_expression-3', 'ambient_expression-30', 'ambient_expression-31', 'ambient_expression-32', 'ambient_expression-33', 'ambient_expression-34', 'ambient_expression-35', 'ambient_expression-36', 'ambient_expression-37', 'ambient_expression-38', 'ambient_expression-39', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7', 'ambient_expression-8', 'ambient_expression-9'
    obsm: 'latent_gene_encoding'
```

Example 8 (python):
```python
adata.obs['combinedID'].value_counts()
```

Example 9 (unknown):
```unknown
BHF_F_Hea13188316_BHF_F_Hea13187618      59866
7089STDY13216924_BHF_F_Hea13242531       40000
BHF_F_Hea11933665_BHF_F_Hea11596618      33997
7089STDY13216925_BHF_F_Hea13242532       30453
BHF_F_Hea13188318_BHF_F_Hea13187620      27856
7089STDY13216920_BHF_F_Hea13242527       27498
BHF_F_Hea13168898_BHF_F_Hea13168514      24781
7089STDY13216927_BHF_F_Hea13242534       24666
7089STDY13216926_BHF_F_Hea13242533       24602
BHF_F_Hea13188324_BHF_F_Hea13187626      24381
7089STDY13216921_BHF_F_Hea13242528       24211
BHF_F_Hea13168896_BHF_F_Hea13168512      22267
HCAHeartST13386009_HCAHeartST13303419    21577
BHF_F_Hea11933666_BHF_F_Hea11596619      21509
BHF_F_Hea13168899_BHF_F_Hea13168515      21418
7089STDY13216922_BHF_F_Hea13242529       21402
BHF_F_Hea13188317_BHF_F_Hea13187619      21327
BHF_F_Hea11933664_BHF_F_Hea11596617      20338
BHF_F_Hea11933667_BHF_F_Hea11596620      19811
BHF_F_Hea13168894_BHF_F_Hea13168510      19258
BHF_F_Hea11933669_BHF_F_Hea11596622      19029
BHF_F_Hea13168897_BHF_F_Hea13168513      18609
HCAHeartST13386010_HCAHeartST13303420    18350
BHF_F_Hea13168895_BHF_F_Hea13168511      18315
BHF_F_Hea11933668_BHF_F_Hea11596621      17494
BHF_F_Hea13168893_BHF_F_Hea13168509      11782
BHF_F_Hea11933670_BHF_F_Hea11596623      11068
BHF_F_Hea11933673_BHF_F_Hea11596626      10769
BHF_F_Hea11933671_BHF_F_Hea11596624      10256
BHF_F_Hea11064672_BHF_F_Hea11031825      10191
BHF_F_Hea13168892_BHF_F_Hea13168508      10023
BHF_F_Hea11933672_BHF_F_Hea11596625       9711
BHF_F_Hea11933674_BHF_F_Hea11596627       6199
BHF_F_Hea13188328_BHF_F_Hea13187630       5340
BHF_F_Hea11064671_BHF_F_Hea11031824       4757
BHF_F_Hea11064670_BHF_F_Hea11031823       2665
BHF_F_Hea13188329_BHF_F_Hea13187631       1878
7089STDY13216923_BHF_F_Hea13242530        1873
BHF_F_Hea13188330_BHF_F_Hea13187632       1750
BHF_F_Hea11933675_BHF_F_Hea11596628        702
Name: combinedID, dtype: int64
```

Example 10 (python):
```python
adata.obs['scrublet_score'].hist(bins=100)
```

Example 11 (unknown):
```unknown
<AxesSubplot:>
```

Example 12 (python):
```python
# set threshold (arbitrary)
dbl_score_thresh=0.15

prop = np.sum(adata.obs['scrublet_score']>=dbl_score_thresh)/len(adata)
print(f'proportion of doublets: {prop}')
```

Example 13 (unknown):
```unknown
proportion of doublets: 0.2872091847546812
```

Example 14 (python):
```python
# Save doublet ids
doublets = adata.obs_names[(adata.obs['scrublet_score']<dbl_score_thresh)==False]
with open('/nfs/team205/heart/anndata_objects/Foetal/QC/Multiome_rm-doublets.txt', 'w') as f:
    for line in doublets:
        f.write(line)
        f.write('\n')

# Remove doublets
adata = adata[adata.obs['scrublet_score']<dbl_score_thresh]
print(adata.shape)
```

Example 15 (unknown):
```unknown
(514620, 36601)
```

Example 16 (python):
```python
sc.pp.filter_cells(adata, min_genes=500, inplace=True)
sc.pp.filter_cells(adata, min_counts=1000, inplace=True)
adata.shape
```

Example 17 (unknown):
```unknown
filtered out 73084 cells that have less than 500 genes expressed
Trying to set attribute `.obs` of view, copying.
filtered out 41920 cells that have less than 1000 counts
```

Example 18 (unknown):
```unknown
(399616, 36601)
```

Example 19 (python):
```python
adata.var['mt'] = adata.var['gene_name'].str.startswith('MT-')  # annotate the group of mitochondrial genes as 'mt'
adata.var['ribo'] = adata.var['gene_name'].str.startswith('RPS','RPL')  # annotate the group of ribosomal genes as 'ribo'
sc.pp.calculate_qc_metrics(adata, qc_vars=['mt','ribo'], percent_top=None, log1p=True, inplace=True)
```

Example 20 (python):
```python
plt.rcParams['figure.figsize'] = [3, 2]

sc.pl.scatter(adata, x='total_counts', y='n_genes')
sc.pl.scatter(adata, x='total_counts', y='pct_counts_mt')
sc.pl.scatter(adata, x='total_counts', y='pct_counts_ribo')
```

Example 21 (python):
```python
plt.rcParams['figure.figsize'] = [2, 2]
adata = adata[adata.obs['pct_counts_mt'] < 5, :]
adata = adata[adata.obs['pct_counts_ribo'] < 5, :]
sc.pl.violin(adata, ['n_genes', 'total_counts', 'log1p_total_counts','pct_counts_mt', 'pct_counts_ribo'],jitter = 0.1, multi_panel = True)
```

Example 22 (python):
```python
sc.settings.set_figure_params(dpi = 120, color_map = 'RdPu')
plt.rcParams['figure.figsize'] = 10,3
sc.pl.violin(adata,keys=['log1p_total_counts'],groupby='sangerID',rotation=90,jitter=False)
```

Example 23 (unknown):
```unknown
Trying to set attribute `.uns` of view, copying.
```

Example 24 (python):
```python
plt.rcParams['figure.figsize'] = 10,3
sc.pl.violin(adata,keys=['pct_counts_mt'],groupby='sangerID',rotation=90,jitter=False)
```

Example 25 (python):
```python
plt.rcParams['figure.figsize'] = 10,3
sc.pl.violin(adata,keys=['pct_counts_ribo'],groupby='sangerID',rotation=90,jitter=False)
```

Example 26 (python):
```python
sc.settings.set_figure_params(dpi = 120, color_map = 'RdPu')
```

Example 27 (python):
```python
adata.shape
```

Example 28 (unknown):
```unknown
(285185, 36601)
```

Example 29 (python):
```python
adata_c = adata.copy()

adata_c.var.set_index('gene_name',inplace=True)
adata_c.var_names = adata_c.var_names.astype('str')
adata_c.var_names_make_unique()

sc.pp.filter_genes(adata_c, min_cells=3)
sc.pp.normalize_total(adata_c, target_sum=1e4)
sc.pp.log1p(adata_c)
```

Example 30 (unknown):
```unknown
filtered out 2031 genes that are detected in less than 3 cells
normalizing counts per cell
    finished (0:00:02)
```

Example 31 (python):
```python
# scoring hemoglobin genes
# HGNC, Hemoglobin subunits, groupID:940
genes=['HBA1','HBA2','HBB','HBD','HBE1','HBG1','HBG2','HBM','HBQ1','HBZ']
sc.tl.score_genes(adata_c,gene_list=genes,score_name='HB_score')
```

Example 32 (python):
```python
computing score 'HB_score'
    finished: added
    'HB_score', score of gene set (adata.obs).
    399 total control genes are used. (0:00:10)
```

Example 33 (python):
```python
len(adata_c.obs['sangerID'].unique())
```

Example 34 (python):
```python
sc.settings.set_figure_params(dpi = 80, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')

fig,axs=plt.subplots(nrows=8,ncols=5,figsize=(15,15))

for i,ID in enumerate(adata_c.obs['sangerID'].unique()):
    sample = adata_c[adata_c.obs['sangerID']==ID]
    axs[i//5,i%5].hist(sample.obs['HB_score'],bins=100)
    axs[i//5,i%5].set_title(ID)

fig.tight_layout()
plt.show()
```

Example 35 (python):
```python
# add HB score to original adata
adata.obs.loc[adata_c.obs_names,'HB_score']=adata_c.obs['HB_score']
```

Example 36 (python):
```python
HBscore_thresh=1

adata=adata[adata.obs['HB_score']<HBscore_thresh]
print(adata.shape)
```

Example 37 (unknown):
```unknown
(274033, 36601)
```

Example 38 (python):
```python
adata.obs['combinedID'].value_counts()
```

Example 39 (unknown):
```unknown
BHF_F_Hea13188316_BHF_F_Hea13187618      21518
BHF_F_Hea13168898_BHF_F_Hea13168514      13895
7089STDY13216920_BHF_F_Hea13242527       12771
BHF_F_Hea13168899_BHF_F_Hea13168515      12369
BHF_F_Hea11933665_BHF_F_Hea11596618      11830
BHF_F_Hea13168896_BHF_F_Hea13168512      11191
BHF_F_Hea11933664_BHF_F_Hea11596617      11022
7089STDY13216921_BHF_F_Hea13242528       10826
BHF_F_Hea11933666_BHF_F_Hea11596619      10463
BHF_F_Hea11933667_BHF_F_Hea11596620      10340
BHF_F_Hea13168894_BHF_F_Hea13168510       9994
BHF_F_Hea11933669_BHF_F_Hea11596622       9539
BHF_F_Hea11933668_BHF_F_Hea11596621       9490
BHF_F_Hea13168895_BHF_F_Hea13168511       9431
BHF_F_Hea13168897_BHF_F_Hea13168513       9315
7089STDY13216924_BHF_F_Hea13242531        8598
BHF_F_Hea13188318_BHF_F_Hea13187620       7474
BHF_F_Hea13168893_BHF_F_Hea13168509       6747
BHF_F_Hea13188324_BHF_F_Hea13187626       6266
7089STDY13216927_BHF_F_Hea13242534        6061
HCAHeartST13386010_HCAHeartST13303420     6015
BHF_F_Hea13168892_BHF_F_Hea13168508       5651
7089STDY13216922_BHF_F_Hea13242529        5382
BHF_F_Hea13188317_BHF_F_Hea13187619       5210
HCAHeartST13386009_HCAHeartST13303419     4833
BHF_F_Hea11064672_BHF_F_Hea11031825       4499
7089STDY13216926_BHF_F_Hea13242533        4474
BHF_F_Hea11933673_BHF_F_Hea11596626       4169
BHF_F_Hea11933671_BHF_F_Hea11596624       3861
BHF_F_Hea11933672_BHF_F_Hea11596625       3858
BHF_F_Hea11933670_BHF_F_Hea11596623       3491
7089STDY13216925_BHF_F_Hea13242532        3475
BHF_F_Hea11064671_BHF_F_Hea11031824       2808
BHF_F_Hea11933674_BHF_F_Hea11596627       2777
BHF_F_Hea13188328_BHF_F_Hea13187630       1489
BHF_F_Hea13188329_BHF_F_Hea13187631       1085
BHF_F_Hea13188330_BHF_F_Hea13187632        837
BHF_F_Hea11064670_BHF_F_Hea11031823        611
BHF_F_Hea11933675_BHF_F_Hea11596628        262
7089STDY13216923_BHF_F_Hea13242530         106
Name: combinedID, dtype: int64
```

Example 40 (python):
```python
# read in SOC-CMO result
demul = pd.read_csv('/nfs/team205/heart/mapped/cellranger-arc200/SOC_out/SOC-CMO-assignment_multiome-foetal_20230207.csv',
                   index_col=0)
print(demul.shape)
print(demul['SOC-CMO_assignment'].value_counts())
demul.head()
```

Example 41 (unknown):
```unknown
(74984, 4)
Hst39               36363
Hst40               21936
donor_unassigned    11437
Hst36                5248
Name: SOC-CMO_assignment, dtype: int64
```

Example 42 (python):
```python
# just checking
# multiplexed samples which passed through QC-filtering above
multiplex_obsnames = adata.obs_names[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                                                   '7089STDY13216921_BHF_F_Hea13242528',
                                                                   '7089STDY13216922_BHF_F_Hea13242529',
                                                                   '7089STDY13216923_BHF_F_Hea13242530'])]
print(demul.loc[multiplex_obsnames].shape)
print(demul.loc[multiplex_obsnames]['SOC-CMO_assignment'].value_counts())

# make 'donor_by_libraryprep' column
adata.obs['donor_by_library-prep'] = adata.obs['donor'].copy()
# make 'multiplexed' column
adata.obs['multiplexed'] = 'No'
adata.obs.loc[multiplex_obsnames,'multiplexed'] = 'Yes'

# replace 'donor' column for multiplexed samples
adata.obs['donor'] = adata.obs['donor'].astype('str')
adata.obs.loc[multiplex_obsnames,'donor'] = demul.loc[multiplex_obsnames,'SOC-CMO_assignment']

# add some SOC metrices
cols_to_add = ['SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet']
adata.obs[cols_to_add] = demul[cols_to_add].loc[multiplex_obsnames].reindex(adata.obs_names)

# check
print(adata.obs.loc[multiplex_obsnames]['donor'].value_counts())
print(adata.shape)
```

Example 43 (unknown):
```unknown
Trying to set attribute `.obs` of view, copying.
```

Example 44 (unknown):
```unknown
(29085, 4)
Hst39               17519
Hst40                8084
Hst36                1787
donor_unassigned     1695
Name: SOC-CMO_assignment, dtype: int64
Hst39               17519
Hst40                8084
Hst36                1787
donor_unassigned     1695
Name: donor, dtype: int64
(274033, 36601)
```

Example 45 (python):
```python
# remove unassigned nuclei
adata = adata[adata.obs['donor'] != 'donor_unassigned']
print(adata.shape)
```

Example 46 (unknown):
```unknown
(272338, 36601)
```

Example 47 (python):
```python
# to see which column needs to be updated
adata.obs[adata.obs['multiplexed']=='Yes'][['sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender']].head(n=3)
```

Example 48 (python):
```python
adata.obs[adata.obs['multiplexed']=='Yes'][['sangerID','donor','region']].value_counts()
```

Example 49 (unknown):
```unknown
sangerID          donor  region                                                       
7089STDY13216920  Hst39  Aorta (Hst39), Nodes (Hst40)                                     8819
7089STDY13216921  Hst39  Aorta (Hst39), Nodes (Hst40)                                     7402
7089STDY13216920  Hst40  Aorta (Hst39), Nodes (Hst40)                                     3321
7089STDY13216921  Hst40  Aorta (Hst39), Nodes (Hst40)                                     2934
7089STDY13216922  Hst40  Heart (Hst36), Heart-wo-nodes (Hst39), Heart-wo-nodes (Hst40)    1828
                  Hst36  Heart (Hst36), Heart-wo-nodes (Hst39), Heart-wo-nodes (Hst40)    1786
                  Hst39  Heart (Hst36), Heart-wo-nodes (Hst39), Heart-wo-nodes (Hst40)    1298
7089STDY13216923  Hst40  Heart (Hst36), Heart-wo-nodes (Hst39), Heart-wo-nodes (Hst40)       1
                  Hst36  Heart (Hst36), Heart-wo-nodes (Hst39), Heart-wo-nodes (Hst40)       1
dtype: int64
```

Example 50 (python):
```python
# prepare metadata dictionary
age_gender = {
    'Hst36': ['PCW-12w','Female'],
    'Hst39': ['PCW-15w','Female'],
    'Hst40': ['PCW-14w','Male'],
}

regions = {
    '7089STDY13216920_Hst39':'Aorta',
    '7089STDY13216920_Hst40':'Nodes',
    '7089STDY13216921_Hst39':'Aorta',
    '7089STDY13216921_Hst40':'Nodes',
    '7089STDY13216922_Hst36':'Heart',
    '7089STDY13216922_Hst39':'Heart-wo-nodes',
    '7089STDY13216922_Hst40':'Heart-wo-nodes',
    '7089STDY13216923_Hst36':'Heart',
    '7089STDY13216923_Hst39':'Heart-wo-nodes',
    '7089STDY13216923_Hst40':'Heart-wo-nodes',
}
```

Example 51 (python):
```python
# update
SangerIDs = list(set(adata.obs[adata.obs['multiplexed']=='Yes']['sangerID']))
Donors = list(set(adata.obs[adata.obs['multiplexed']=='Yes']['donor']))

for sID in SangerIDs:
    for d in Donors:
        obsnames_to_update = adata.obs_names[(adata.obs['sangerID']==sID)&(adata.obs['donor']==d)]
        print(f'{sID}, {d}: {len(obsnames_to_update)} nuclei')
        # update
        if len(obsnames_to_update)>0:
            adata.obs.loc[obsnames_to_update,'region'] = regions[f'{sID}_{d}']
            adata.obs.loc[obsnames_to_update,'region_finest'] = regions[f'{sID}_{d}']
            adata.obs.loc[obsnames_to_update,'age'] = age_gender[d][0]
            adata.obs.loc[obsnames_to_update,'gender'] = age_gender[d][1]
        else:
            pass
        del obsnames_to_update
```

Example 52 (unknown):
```unknown
7089STDY13216921, Hst36: 0 nuclei
7089STDY13216921, Hst40: 2934 nuclei
7089STDY13216921, Hst39: 7402 nuclei
7089STDY13216922, Hst36: 1786 nuclei
7089STDY13216922, Hst40: 1828 nuclei
7089STDY13216922, Hst39: 1298 nuclei
7089STDY13216923, Hst36: 1 nuclei
7089STDY13216923, Hst40: 1 nuclei
7089STDY13216923, Hst39: 0 nuclei
7089STDY13216920, Hst36: 0 nuclei
7089STDY13216920, Hst40: 3321 nuclei
7089STDY13216920, Hst39: 8819 nuclei
```

Example 53 (python):
```python
# save
adata.write('/nfs/team205/heart/anndata_objects/Foetal/QC/multiome_RNA_foetal.h5ad')
adata
```

Example 54 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/anndata/_core/anndata.py:1207: ImplicitModificationWarning: Initializing view as actual.
  warnings.warn(
Trying to set attribute `.obs` of view, copying.
... storing 'donor' as categorical
Trying to set attribute `.obs` of view, copying.
... storing 'multiplexed' as categorical
Trying to set attribute `.obs` of view, copying.
... storing 'SOC | status' as categorical
```

Example 55 (unknown):
```unknown
AnnData object with n_obs × n_vars = 272338 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score', 'donor_by_library-prep', 'multiplexed', 'SOC | status', 'SOC | log_prob_singleton', 'SOC | log_prob_doublet'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-10', 'ambient_expression-11', 'ambient_expression-12', 'ambient_expression-13', 'ambient_expression-14', 'ambient_expression-15', 'ambient_expression-16', 'ambient_expression-17', 'ambient_expression-18', 'ambient_expression-19', 'ambient_expression-2', 'ambient_expression-20', 'ambient_expression-21', 'ambient_expression-22', 'ambient_expression-23', 'ambient_expression-24', 'ambient_expression-25', 'ambient_expression-26', 'ambient_expression-27', 'ambient_expression-28', 'ambient_expression-29', 'ambient_expression-3', 'ambient_expression-30', 'ambient_expression-31', 'ambient_expression-32', 'ambient_expression-33', 'ambient_expression-34', 'ambient_expression-35', 'ambient_expression-36', 'ambient_expression-37', 'ambient_expression-38', 'ambient_expression-39', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7', 'ambient_expression-8', 'ambient_expression-9', 'mt', 'ribo', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
    uns: 'sangerID_colors'
    obsm: 'latent_gene_encoding'
```

Example 56 (python):
```python
adata.obs['combinedID'].value_counts()
```

Example 57 (unknown):
```unknown
BHF_F_Hea13188316_BHF_F_Hea13187618      21518
BHF_F_Hea13168898_BHF_F_Hea13168514      13895
BHF_F_Hea13168899_BHF_F_Hea13168515      12369
7089STDY13216920_BHF_F_Hea13242527       12140
BHF_F_Hea11933665_BHF_F_Hea11596618      11830
BHF_F_Hea13168896_BHF_F_Hea13168512      11191
BHF_F_Hea11933664_BHF_F_Hea11596617      11022
BHF_F_Hea11933666_BHF_F_Hea11596619      10463
BHF_F_Hea11933667_BHF_F_Hea11596620      10340
7089STDY13216921_BHF_F_Hea13242528       10336
BHF_F_Hea13168894_BHF_F_Hea13168510       9994
BHF_F_Hea11933669_BHF_F_Hea11596622       9539
BHF_F_Hea11933668_BHF_F_Hea11596621       9490
BHF_F_Hea13168895_BHF_F_Hea13168511       9431
BHF_F_Hea13168897_BHF_F_Hea13168513       9315
7089STDY13216924_BHF_F_Hea13242531        8598
BHF_F_Hea13188318_BHF_F_Hea13187620       7474
BHF_F_Hea13168893_BHF_F_Hea13168509       6747
BHF_F_Hea13188324_BHF_F_Hea13187626       6266
7089STDY13216927_BHF_F_Hea13242534        6061
HCAHeartST13386010_HCAHeartST13303420     6015
BHF_F_Hea13168892_BHF_F_Hea13168508       5651
BHF_F_Hea13188317_BHF_F_Hea13187619       5210
7089STDY13216922_BHF_F_Hea13242529        4912
HCAHeartST13386009_HCAHeartST13303419     4833
BHF_F_Hea11064672_BHF_F_Hea11031825       4499
7089STDY13216926_BHF_F_Hea13242533        4474
BHF_F_Hea11933673_BHF_F_Hea11596626       4169
BHF_F_Hea11933671_BHF_F_Hea11596624       3861
BHF_F_Hea11933672_BHF_F_Hea11596625       3858
BHF_F_Hea11933670_BHF_F_Hea11596623       3491
7089STDY13216925_BHF_F_Hea13242532        3475
BHF_F_Hea11064671_BHF_F_Hea11031824       2808
BHF_F_Hea11933674_BHF_F_Hea11596627       2777
BHF_F_Hea13188328_BHF_F_Hea13187630       1489
BHF_F_Hea13188329_BHF_F_Hea13187631       1085
BHF_F_Hea13188330_BHF_F_Hea13187632        837
BHF_F_Hea11064670_BHF_F_Hea11031823        611
BHF_F_Hea11933675_BHF_F_Hea11596628        262
7089STDY13216923_BHF_F_Hea13242530           2
Name: combinedID, dtype: int64
```

Example 58 (python):
```python
adata.obs['donor'].value_counts()
```

Example 59 (unknown):
```unknown
Hst41    129909
Hst33     62684
Hst39     30591
C92       15379
Hst40     11559
Hst36      7848
C87        4499
C104       3411
C98        3039
C85        2808
C82         611
Name: donor, dtype: int64
```

Example 60 (python):
```python
pd.crosstab(adata.obs['donor'],adata.obs['age'])
```

Example 61 (python):
```python
pd.crosstab(adata.obs['donor'],adata.obs['gender'])
```

Example 62 (python):
```python
adata.obs['SOC | status'].value_counts()
```

Example 63 (javascript):
```javascript
nan           244948
singlet        27068
doublet          246
unassigned        76
Name: SOC | status, dtype: int64
```

Example 64 (python):
```python
adata.obs[adata.obs['multiplexed']=='Yes']['donor_by_library-prep'].value_counts()
```

Example 65 (unknown):
```unknown
Hst39, Hst40           22476
Hst36, Hst39, Hst40     4914
Hst41                      0
Hst40                      0
Hst39                      0
Hst36                      0
Hst33                      0
C104                       0
C98                        0
C92                        0
C87                        0
C85                        0
C82                        0
Name: donor_by_library-prep, dtype: int64
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/QC/Foetal_Multiome-RNA_No1_make-anndata_run-scrublet.html

**Contents:**
- Import modules¶
- Functions¶
- Create anndata object¶
- Run scrublet¶
- Save¶

721979 rows × 21 columns

**Examples:**

Example 1 (python):
```python
import numpy as np
import pandas as pd
import scanpy as sc
import matplotlib.pyplot as plt
import scrublet as scr
import session_info
```

Example 2 (python):
```python
sc.settings.verbosity = 3
sc.settings.set_figure_params(dpi = 160, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.3.4
numpy               1.20.1
pandas              1.1.4
scanpy              1.7.1
scipy               1.6.1
scrublet            NA
session_info        1.0.0
statsmodels         0.12.2
tables              3.6.1
-----
```

Example 4 (unknown):
```unknown
PIL                 8.1.2
anndata             0.7.5
anyio               NA
attr                20.3.0
babel               2.9.0
backcall            0.2.0
brotli              NA
cairo               1.20.0
certifi             2020.12.05
cffi                1.14.5
chardet             4.0.0
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.0
dask                2021.03.1
dateutil            2.8.1
decorator           4.4.2
fsspec              0.8.7
get_version         2.1
google              NA
h5py                2.10.0
idna                2.10
igraph              0.8.3
ipykernel           5.5.0
ipython_genutils    0.2.0
jedi                0.18.0
jinja2              2.11.3
joblib              1.0.1
json5               NA
jsonschema          3.2.0
jupyter_server      1.4.1
jupyterlab_server   2.3.0
kiwisolver          1.3.1
legacy_api_wrap     0.0.0
leidenalg           0.8.3
llvmlite            0.34.0
louvain             0.7.0
markupsafe          1.1.1
mpl_toolkits        NA
natsort             7.1.1
nbclassic           NA
nbformat            5.1.2
numba               0.51.2
numexpr             2.7.3
packaging           20.9
parso               0.8.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
prometheus_client   NA
prompt_toolkit      3.0.16
psutil              5.8.0
ptyprocess          0.7.0
pvectorc            NA
pyarrow             0.16.0
pygments            2.8.0
pyparsing           2.4.7
pyrsistent          NA
pytoml              NA
pytz                2021.1
requests            2.25.1
ruamel              NA
send2trash          NA
setuptools_scm      NA
simplejson          3.17.2
sinfo               0.3.1
six                 1.15.0
sklearn             0.22
sniffio             1.2.0
socks               1.7.1
sphinxcontrib       NA
storemagic          NA
tblib               1.7.0
texttable           1.6.3
tlz                 0.11.0
toolz               0.11.1
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
uritemplate         3.0.1
urllib3             1.26.3
wcwidth             0.2.5
yaml                5.1.2
zmq                 22.0.3
```

Example 5 (unknown):
```unknown
-----
IPython             7.21.0
jupyter_client      6.1.11
jupyter_core        4.7.1
jupyterlab          3.0.9
notebook            6.2.0
-----
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0]
Linux-4.15.0-112-generic-x86_64-with-glibc2.10
-----
Session information updated at 2023-02-08 21:30
```

Example 6 (python):
```python
"""Functions for downstream work with outputs of remove-background."""

# to read cellbender outputs<br>
# https://github.com/broadinstitute/CellBender/issues/57

import tables
import numpy as np
import scipy.sparse as sp
from typing import Dict


def dict_from_h5(file: str) -> Dict[str, np.ndarray]:
    """Read in everything from an h5 file and put into a dictionary."""
    d = {}
    with tables.open_file(file) as f:
        # read in everything
        for array in f.walk_nodes("/", "Array"):
            d[array.name] = array.read()
    return d


def anndata_from_h5(file: str,
                    analyzed_barcodes_only: bool = True) -> 'anndata.AnnData':
    """Load an output h5 file into an AnnData object for downstream work.

    Args:
        file: The h5 file
        analyzed_barcodes_only: False to load all barcodes, so that the size of
            the AnnData object will match the size of the input raw count matrix.
            True to load a limited set of barcodes: only those analyzed by the
            algorithm. This allows relevant latent variables to be loaded
            properly into adata.obs and adata.obsm, rather than adata.uns.

    Returns:
        adata: The anndata object, populated with inferred latent variables
            and metadata.

    """

    try:
        import anndata
    except ImportError:
        raise ImportError('The anndata package must be installed to use the '
                          'function anndata_from_h5()')

    d = dict_from_h5(file)
    X = sp.csc_matrix((d.pop('data'), d.pop('indices'), d.pop('indptr')),
                      shape=d.pop('shape')).transpose().tocsr()

    if analyzed_barcodes_only:
        if 'barcodes_analyzed_inds' in d.keys():
            X = X[d['barcodes_analyzed_inds'], :]
            d['barcodes'] = d['barcodes'][d['barcodes_analyzed_inds']]
        elif 'barcode_indices_for_latents' in d.keys():
            X = X[d['barcode_indices_for_latents'], :]
            d['barcodes'] = d['barcodes'][d['barcode_indices_for_latents']]
        else:
            print('Warning: analyzed_barcodes_only=True, but the key '
                  '"barcodes_analyzed_inds" or "barcode_indices_for_latents" '
                  'is missing from the h5 file. '
                  'Will output all barcodes, and proceed as if '
                  'analyzed_barcodes_only=False')

    # Construct the count matrix.
    adata = anndata.AnnData(X=X,
                            obs={'barcode': d.pop('barcodes').astype(str)},
                            var={'gene_name': (d.pop('gene_names') if 'gene_names' in d.keys()
                                               else d.pop('name')).astype(str)})
    adata.obs.set_index('barcode', inplace=True)
    adata.var.set_index('gene_name', inplace=True)

    # Add other information to the adata object in the appropriate slot.
    for key, value in d.items():
        try:
            value = np.asarray(value)
            if len(value.shape) == 0:
                adata.uns[key] = value
            elif value.shape[0] == X.shape[0]:
                if (len(value.shape) < 2) or (value.shape[1] < 2):
                    adata.obs[key] = value
                else:
                    adata.obsm[key] = value
            elif value.shape[0] == X.shape[1]:
                if value.dtype.name.startswith('bytes'):
                    adata.var[key] = value.astype(str)
                else:
                    adata.var[key] = value
            else:
                adata.uns[key] = value
        except Exception:
            print('Unable to load data into AnnData: ', key, value, type(value))

    if analyzed_barcodes_only:
        for col in adata.obs.columns[adata.obs.columns.str.startswith('barcodes_analyzed')
                                     | adata.obs.columns.str.startswith('barcode_indices')]:
            try:
                del adata.obs[col]
            except Exception:
                pass

    return adata
```

Example 7 (python):
```python
# from https://github.com/Teichlab/mapcloud/blob/master/scripts/starsolo/postprocess.py

from statsmodels.stats.multitest import multipletests
# from emptydrops.matrix import CountMatrix
# from emptydrops import find_nonambient_barcodes
import scrublet as scr
import scanpy as sc
import pandas as pd
import numpy as np
import scipy

#some functions that Ni uses in scanpy scripts to run scrublet
#which in turn are inspired by my original notebook on the matter
#(extracted from scanpy_scripts 0.2.10 to get around scanpy version incompatibility)
def test_outlier(x, upper_mad_only=True):
	med = np.median(x)
	if upper_mad_only:
		mad = np.median(x[x>med] - med) * 1.4826
	else:
		mad = np.median(np.abs(x - med)) * 1.4826
	pvals = 1 - scipy.stats.norm.cdf(x, loc=med, scale=mad)
	bh_pvals = multipletests(pvals, method='fdr_bh')[1]
	return pvals, bh_pvals

def run_scrublet(adata, resolution_function=None):
	old_verbosity = sc.settings.verbosity
	sc.settings.verbosity = 1
	if resolution_function is None:
		resolution_function = lambda x: np.maximum(np.maximum(np.log10(x)-1, 0)**2, 0.1)
	scrub = scr.Scrublet(adata.X)
	#this has the potential to brick for poor quality data
	#if so, abort it and everything downstream
	try:
		ds, pd = scrub.scrub_doublets(verbose=False)
	except:
		return
	adata.obs['scrublet_score'] = ds

	adata_copy = adata.copy()
	sc.pp.filter_genes(adata_copy, min_cells=3)
	sc.pp.normalize_total(adata_copy, target_sum=1e4)
	sc.pp.log1p(adata_copy)
	sc.pp.highly_variable_genes(adata_copy, min_mean=0.0125, max_mean=3, min_disp=0.5, subset=True)
	sc.pp.scale(adata_copy, zero_center=False)
	sc.pp.pca(adata_copy, svd_solver='arpack', zero_center=False)
	sc.pp.neighbors(adata_copy, n_pcs=30)
	sc.tl.umap(adata_copy)
	sc.tl.leiden(adata_copy, resolution=1)
	for clst in np.unique(adata_copy.obs['leiden']):
		clst_size = sum(adata_copy.obs['leiden'] == clst)
		sc.tl.leiden(adata_copy, restrict_to=('leiden', [clst]), resolution=resolution_function(clst_size), key_added='leiden_R')
		adata_copy.obs['leiden'] = adata_copy.obs['leiden_R']
	clst_meds = []
	for clst in np.unique(adata_copy.obs['leiden']):
		k = adata_copy.obs['leiden'] == clst
		clst_med = np.median(adata_copy.obs.loc[k, 'scrublet_score'])
		adata_copy.obs.loc[k, 'cluster_scrublet_score'] = clst_med
		clst_meds.append(clst_med)
	clst_meds = np.array(clst_meds)
	pvals, bh_pvals = test_outlier(clst_meds)
	for i, clst in enumerate(np.unique(adata_copy.obs['leiden'])):
		k = adata_copy.obs['leiden'] == clst
		adata_copy.obs.loc[k, 'pval'] = pvals[i]
		adata_copy.obs.loc[k, 'bh_pval'] = bh_pvals[i]
	sc.settings.verbosity = old_verbosity
	#need to also export the clustering, for soupx purposes
	adata.obs['scrublet_leiden'] = adata_copy.obs['leiden']
	adata.obs['scrublet_score'] = adata_copy.obs['scrublet_score']
	adata.obs['cluster_scrublet_score'] = adata_copy.obs['cluster_scrublet_score']
	adata.obs['doublet_pval'] = adata_copy.obs['pval']
	adata.obs['doublet_bh_pval'] = adata_copy.obs['bh_pval']
	del adata_copy
```

Example 8 (python):
```python
# read in metadata
metadata = pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230208.csv', sep = ',', index_col = None)
metadata = metadata[metadata['Publication']=='Foetal']

# select healthy data
metadata = metadata[metadata['status']=='Healthy']


print(metadata['modality'].value_counts())
```

Example 9 (unknown):
```unknown
Multiome-RNA     40
Multiome-ATAC    40
Visium           23
scRNA            20
Visium-FFPE       2
Name: modality, dtype: int64
```

Example 10 (python):
```python
# select modality
metadata = metadata[metadata['modality']=='Multiome-RNA']
```

Example 11 (python):
```python
########### editted: put id to var_names ###########

# read in
adatas=[]
for i in range(len(metadata)):
    print(metadata.iloc[i]['combinedID'])
    
    path=metadata.iloc[i]['CellBender_out']
    adatas.append(anndata_from_h5(path + '/' + path.split('/')[-1] + '_cellbender_out_filtered.h5', analyzed_barcodes_only=False))
    del path
    
    # replace var_names with ensembleID
    adatas[i].var.reset_index(inplace=True)
    if 'id' in adatas[i].var.columns:
        adatas[i].var.set_index('id',inplace=True)
    else:
        adatas[i].var.set_index('genes',inplace=True)
    
    # modify barcodes
    adatas[i].obs.index = metadata.iloc[i]['combinedID'] + '_' + adatas[i].obs.index 
    
    # add metadata
    for col in ['sangerID','combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age',
                'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x','flushed']:
        adatas[i].obs[col] = metadata.iloc[i][col]


# concatenate
adata = adatas[0].concatenate(adatas[1:], index_unique = None, batch_key=None)
adata.shape
```

Example 12 (unknown):
```unknown
BHF_F_Hea11064670_BHF_F_Hea11031823
BHF_F_Hea11064671_BHF_F_Hea11031824
BHF_F_Hea11064672_BHF_F_Hea11031825
BHF_F_Hea11933664_BHF_F_Hea11596617
BHF_F_Hea11933665_BHF_F_Hea11596618
BHF_F_Hea11933666_BHF_F_Hea11596619
BHF_F_Hea11933667_BHF_F_Hea11596620
BHF_F_Hea11933668_BHF_F_Hea11596621
BHF_F_Hea11933669_BHF_F_Hea11596622
BHF_F_Hea11933670_BHF_F_Hea11596623
BHF_F_Hea11933671_BHF_F_Hea11596624
BHF_F_Hea11933672_BHF_F_Hea11596625
BHF_F_Hea11933673_BHF_F_Hea11596626
BHF_F_Hea11933674_BHF_F_Hea11596627
BHF_F_Hea11933675_BHF_F_Hea11596628
BHF_F_Hea13188316_BHF_F_Hea13187618
BHF_F_Hea13188317_BHF_F_Hea13187619
BHF_F_Hea13188318_BHF_F_Hea13187620
BHF_F_Hea13188324_BHF_F_Hea13187626
BHF_F_Hea13188328_BHF_F_Hea13187630
BHF_F_Hea13188329_BHF_F_Hea13187631
BHF_F_Hea13188330_BHF_F_Hea13187632
BHF_F_Hea13168892_BHF_F_Hea13168508
BHF_F_Hea13168893_BHF_F_Hea13168509
BHF_F_Hea13168894_BHF_F_Hea13168510
BHF_F_Hea13168895_BHF_F_Hea13168511
BHF_F_Hea13168896_BHF_F_Hea13168512
BHF_F_Hea13168897_BHF_F_Hea13168513
BHF_F_Hea13168898_BHF_F_Hea13168514
BHF_F_Hea13168899_BHF_F_Hea13168515
7089STDY13216920_BHF_F_Hea13242527
7089STDY13216921_BHF_F_Hea13242528
7089STDY13216922_BHF_F_Hea13242529
7089STDY13216923_BHF_F_Hea13242530
7089STDY13216924_BHF_F_Hea13242531
7089STDY13216925_BHF_F_Hea13242532
7089STDY13216926_BHF_F_Hea13242533
7089STDY13216927_BHF_F_Hea13242534
HCAHeartST13386009_HCAHeartST13303419
HCAHeartST13386010_HCAHeartST13303420
```

Example 13 (unknown):
```unknown
(721979, 36601)
```

Example 14 (python):
```python
for i in range(len(adatas)):
    print(adatas[i].var.columns)
```

Example 15 (unknown):
```unknown
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
```

Example 16 (python):
```python
adata.obs['combinedID'].value_counts()
```

Example 17 (unknown):
```unknown
BHF_F_Hea13188316_BHF_F_Hea13187618      59866
7089STDY13216924_BHF_F_Hea13242531       40000
BHF_F_Hea11933665_BHF_F_Hea11596618      33997
7089STDY13216925_BHF_F_Hea13242532       30453
BHF_F_Hea13188318_BHF_F_Hea13187620      27856
7089STDY13216920_BHF_F_Hea13242527       27498
BHF_F_Hea13168898_BHF_F_Hea13168514      24781
7089STDY13216927_BHF_F_Hea13242534       24666
7089STDY13216926_BHF_F_Hea13242533       24602
BHF_F_Hea13188324_BHF_F_Hea13187626      24381
7089STDY13216921_BHF_F_Hea13242528       24211
BHF_F_Hea13168896_BHF_F_Hea13168512      22267
HCAHeartST13386009_HCAHeartST13303419    21577
BHF_F_Hea11933666_BHF_F_Hea11596619      21509
BHF_F_Hea13168899_BHF_F_Hea13168515      21418
7089STDY13216922_BHF_F_Hea13242529       21402
BHF_F_Hea13188317_BHF_F_Hea13187619      21327
BHF_F_Hea11933664_BHF_F_Hea11596617      20338
BHF_F_Hea11933667_BHF_F_Hea11596620      19811
BHF_F_Hea13168894_BHF_F_Hea13168510      19258
BHF_F_Hea11933669_BHF_F_Hea11596622      19029
BHF_F_Hea13168897_BHF_F_Hea13168513      18609
HCAHeartST13386010_HCAHeartST13303420    18350
BHF_F_Hea13168895_BHF_F_Hea13168511      18315
BHF_F_Hea11933668_BHF_F_Hea11596621      17494
BHF_F_Hea13168893_BHF_F_Hea13168509      11782
BHF_F_Hea11933670_BHF_F_Hea11596623      11068
BHF_F_Hea11933673_BHF_F_Hea11596626      10769
BHF_F_Hea11933671_BHF_F_Hea11596624      10256
BHF_F_Hea11064672_BHF_F_Hea11031825      10191
BHF_F_Hea13168892_BHF_F_Hea13168508      10023
BHF_F_Hea11933672_BHF_F_Hea11596625       9711
BHF_F_Hea11933674_BHF_F_Hea11596627       6199
BHF_F_Hea13188328_BHF_F_Hea13187630       5340
BHF_F_Hea11064671_BHF_F_Hea11031824       4757
BHF_F_Hea11064670_BHF_F_Hea11031823       2665
BHF_F_Hea13188329_BHF_F_Hea13187631       1878
7089STDY13216923_BHF_F_Hea13242530        1873
BHF_F_Hea13188330_BHF_F_Hea13187632       1750
BHF_F_Hea11933675_BHF_F_Hea11596628        702
Name: combinedID, dtype: int64
```

Example 18 (python):
```python
%%time

# per rxn
for i,ID in enumerate(adata.obs['combinedID'].unique()):
    print(ID)
    
    ad = adata[adata.obs['combinedID'] == ID].copy()
    run_scrublet(ad)
    if i==0:
        meta = ad.obs
    else:
        meta = pd.concat([meta, ad.obs])
    del ad
```

Example 19 (unknown):
```unknown
BHF_F_Hea11064670_BHF_F_Hea11031823
```

Example 20 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 21 (unknown):
```unknown
BHF_F_Hea11064671_BHF_F_Hea11031824
BHF_F_Hea11064672_BHF_F_Hea11031825
```

Example 22 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 23 (unknown):
```unknown
BHF_F_Hea11933664_BHF_F_Hea11596617
```

Example 24 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 25 (unknown):
```unknown
BHF_F_Hea11933665_BHF_F_Hea11596618
```

Example 26 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 27 (unknown):
```unknown
BHF_F_Hea11933666_BHF_F_Hea11596619
```

Example 28 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 29 (unknown):
```unknown
BHF_F_Hea11933667_BHF_F_Hea11596620
```

Example 30 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 31 (unknown):
```unknown
BHF_F_Hea11933668_BHF_F_Hea11596621
```

Example 32 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 33 (unknown):
```unknown
BHF_F_Hea11933669_BHF_F_Hea11596622
```

Example 34 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 35 (unknown):
```unknown
BHF_F_Hea11933670_BHF_F_Hea11596623
```

Example 36 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 37 (unknown):
```unknown
BHF_F_Hea11933671_BHF_F_Hea11596624
```

Example 38 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 39 (unknown):
```unknown
BHF_F_Hea11933672_BHF_F_Hea11596625
```

Example 40 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 41 (unknown):
```unknown
BHF_F_Hea11933673_BHF_F_Hea11596626
```

Example 42 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 43 (unknown):
```unknown
BHF_F_Hea11933674_BHF_F_Hea11596627
```

Example 44 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 45 (unknown):
```unknown
BHF_F_Hea11933675_BHF_F_Hea11596628
BHF_F_Hea13188316_BHF_F_Hea13187618
```

Example 46 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 47 (unknown):
```unknown
BHF_F_Hea13188317_BHF_F_Hea13187619
```

Example 48 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 49 (unknown):
```unknown
BHF_F_Hea13188318_BHF_F_Hea13187620
```

Example 50 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 51 (unknown):
```unknown
BHF_F_Hea13188324_BHF_F_Hea13187626
```

Example 52 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 53 (unknown):
```unknown
BHF_F_Hea13188328_BHF_F_Hea13187630
```

Example 54 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 55 (unknown):
```unknown
BHF_F_Hea13188329_BHF_F_Hea13187631
```

Example 56 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 57 (unknown):
```unknown
BHF_F_Hea13188330_BHF_F_Hea13187632
```

Example 58 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 59 (unknown):
```unknown
BHF_F_Hea13168892_BHF_F_Hea13168508
```

Example 60 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 61 (unknown):
```unknown
BHF_F_Hea13168893_BHF_F_Hea13168509
```

Example 62 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 63 (unknown):
```unknown
BHF_F_Hea13168894_BHF_F_Hea13168510
```

Example 64 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 65 (unknown):
```unknown
BHF_F_Hea13168895_BHF_F_Hea13168511
```

Example 66 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 67 (unknown):
```unknown
BHF_F_Hea13168896_BHF_F_Hea13168512
```

Example 68 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 69 (unknown):
```unknown
BHF_F_Hea13168897_BHF_F_Hea13168513
```

Example 70 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 71 (unknown):
```unknown
BHF_F_Hea13168898_BHF_F_Hea13168514
```

Example 72 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 73 (unknown):
```unknown
BHF_F_Hea13168899_BHF_F_Hea13168515
```

Example 74 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 75 (unknown):
```unknown
7089STDY13216920_BHF_F_Hea13242527
```

Example 76 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 77 (unknown):
```unknown
7089STDY13216921_BHF_F_Hea13242528
```

Example 78 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 79 (unknown):
```unknown
7089STDY13216922_BHF_F_Hea13242529
```

Example 80 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 81 (unknown):
```unknown
7089STDY13216923_BHF_F_Hea13242530
7089STDY13216924_BHF_F_Hea13242531
7089STDY13216925_BHF_F_Hea13242532
```

Example 82 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 83 (unknown):
```unknown
7089STDY13216926_BHF_F_Hea13242533
```

Example 84 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 85 (unknown):
```unknown
7089STDY13216927_BHF_F_Hea13242534
```

Example 86 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 87 (unknown):
```unknown
HCAHeartST13386009_HCAHeartST13303419
```

Example 88 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 89 (unknown):
```unknown
HCAHeartST13386010_HCAHeartST13303420
```

Example 90 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 91 (unknown):
```unknown
CPU times: user 2h 18min 29s, sys: 1h 16min 7s, total: 3h 34min 36s
Wall time: 59min 16s
```

Example 92 (python):
```python
meta_scrub = meta[['scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval']].copy()
meta_scrub.shape
```

Example 93 (unknown):
```unknown
(721979, 5)
```

Example 94 (python):
```python
if meta_scrub.reindex(adata.obs.index).index.equals(adata.obs.index):
    adata.obs = pd.concat([adata.obs, meta_scrub.reindex(adata.obs.index)], axis=1)
else:
    raise Exception('Different barcodes in meta and adata')
```

Example 95 (python):
```python
adata.write('/nfs/team205/heart/anndata_objects/Foetal/QC/multiome_RNA_foetal_prefilter.h5ad')
```

Example 96 (unknown):
```unknown
... storing 'sangerID' as categorical
... storing 'combinedID' as categorical
... storing 'donor' as categorical
... storing 'donor_type' as categorical
... storing 'region' as categorical
... storing 'region_finest' as categorical
... storing 'age' as categorical
... storing 'gender' as categorical
... storing 'facility' as categorical
... storing 'cell_or_nuclei' as categorical
... storing 'modality' as categorical
... storing 'kit_10x' as categorical
... storing 'flushed' as categorical
... storing 'scrublet_leiden' as categorical
... storing 'gene_name' as categorical
... storing 'feature_type' as categorical
```

Example 97 (python):
```python
sc.pl.violin(adata,
             ['scrublet_score', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval'],jitter = 0.2, multi_panel = True)
```

Example 98 (python):
```python
adata.obs['donor'].value_counts()
```

Example 99 (unknown):
```unknown
Hst41                  319810
Hst33                  132178
Hst39                   64602
Hst39, Hst40            51709
C92                     41804
Hst40                   30453
Hst36                   24666
Hst36, Hst39, Hst40     23275
C87                     10191
C104                     8968
C98                      6901
C85                      4757
C82                      2665
Name: donor, dtype: int64
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/VisiumHD-8um_QC-filtering.html

**Contents:**
- Import required modules¶
- Read in 8um bin data¶
- QC and preprocessing¶
- Filtering¶
- Save¶

311232 rows × 14 columns

**Examples:**

Example 1 (python):
```python
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.filterwarnings('ignore')

import scanpy as sc
import anndata
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import seaborn as sns
import os
```

Example 2 (python):
```python
import importlib.util
import sys
spec = importlib.util.spec_from_file_location("module.name", "/rfs/project/rfs-iCNyzSAaucw/kk837/function/python/utils.py")
utils = importlib.util.module_from_spec(spec)
sys.modules["module.name"] = utils
spec.loader.exec_module(utils)
```

Example 3 (python):
```python
sc.settings.verbosity = 1          # verbosity: errors (0), warnings (1), info (2), hints (3)
sc.settings.set_figure_params(dpi=80)
```

Example 4 (unknown):
```unknown
!ls /rfs/project/rfs-iCNyzSAaucw/jc2226/heart_team_data/mapped/spaceranger301
```

Example 5 (unknown):
```unknown
HEA_AD14880300	HEA_AD14880301	HEA_FOET14880396  HEA_FOET14880397
```

Example 6 (python):
```python
# read in object which has window assignment
sample_id = 'HEA_FOET14880396'
path_adata = f"/rfs/project/rfs-iCNyzSAaucw/jc2226/heart_team_data/mapped/spaceranger301/{sample_id}/outs/binned_outputs/square_008um"
adata = sc.read_10x_h5(f'{path_adata}/filtered_feature_bc_matrix.h5')
adata.var_names_make_unique()
adata
```

Example 7 (unknown):
```unknown
AnnData object with n_obs × n_vars = 320061 × 18085
    var: 'gene_ids', 'feature_types', 'genome'
```

Example 8 (python):
```python
adata.obs.head()
```

Example 9 (python):
```python
adata.var.head()
```

Example 10 (python):
```python
adata.var["mt"] = adata.var_names.str.startswith("MT-")
sc.pp.calculate_qc_metrics(adata, qc_vars=["mt"], inplace=True)
```

Example 11 (python):
```python
fig, axs = plt.subplots(1, 4, figsize=(15, 3))
sns.histplot(adata.obs["total_counts"], kde=False, ax=axs[0])
sns.histplot(
    adata.obs["total_counts"][adata.obs["total_counts"] < 1000],
    kde=False,
    bins=40,
    ax=axs[1],
)
sns.histplot(adata.obs["n_genes_by_counts"], kde=False, bins=60, ax=axs[2])
sns.histplot(
    adata.obs["n_genes_by_counts"][adata.obs["n_genes_by_counts"] < 1000],
    kde=False,
    bins=60,
    ax=axs[3],
)
fig.suptitle('8um')
fig.tight_layout()
```

Example 12 (python):
```python
fig, axs = plt.subplots(1, 2, figsize=(10, 3))
sns.histplot(
    adata.obs["total_counts"][adata.obs["total_counts"] < 8000],
    kde=False,
    bins=60,
    ax=axs[0],
)
sns.histplot(
    adata.obs["n_genes_by_counts"][adata.obs["n_genes_by_counts"] < 3500],
    kde=False,
    bins=60,
    ax=axs[1],
)
fig.suptitle('8um')
fig.tight_layout()
```

Example 13 (python):
```python
# mito
sns.histplot(adata.obs['pct_counts_mt'], kde=False)
plt.title('8um')
plt.show()
```

Example 14 (python):
```python
# filtering out poor quality cells/bins
sc.pp.filter_cells(adata, min_genes=100)
print(f'shape of 8um: {adata.shape}')
```

Example 15 (unknown):
```unknown
shape of 8um: (311232, 18085)
```

Example 16 (python):
```python
# prepare coordinates
adata.obs['binned_8_row'] = [int(x.split('_')[2]) for x in adata.obs_names]
adata.obs['binned_8_col'] = [int(x.split('_')[3].replace('-1','')) for x in adata.obs_names]
sns.scatterplot(x='binned_8_col',y='binned_8_row',data=adata.obs)
plt.show()
adata.obs
```

Example 17 (python):
```python
adata.write(f'/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/VisiumHD/{sample_id}_8um-bins_filtered.h5ad')
```

Example 18 (python):
```python
print(f'/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/VisiumHD/{sample_id}_8um-bins_filtered.h5ad')
```

Example 19 (unknown):
```unknown
/rfs/project/rfs-iCNyzSAaucw/kk837/data_objects/Foetal/VisiumHD/HEA_FOET14880396_8um-bins_filtered.h5ad
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/QC/Foetal_T21-Multiome-RNA_No2_QC-filter.html

**Contents:**
- Import modules¶
- Functions¶
- Read in anndata¶
- Remove doublets¶
- Remove low quality cells¶
- Compute QC stats¶
- Remove cells exceeding the ribo/mito threshold¶
- Calculate RBC score¶
- Remove RBCs¶
- Save post-filtered anndata¶
- Just checking¶

**Examples:**

Example 1 (python):
```python
import numpy as np
import pandas as pd
import scanpy as sc
import matplotlib.pyplot as plt
import scrublet as scr
import session_info
```

Example 2 (python):
```python
sc.settings.verbosity = 3
sc.settings.set_figure_params(dpi = 160, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.3.4
numpy               1.20.1
pandas              1.1.4
scanpy              1.7.1
scrublet            NA
session_info        1.0.0
-----
```

Example 4 (unknown):
```unknown
PIL                 8.1.2
anndata             0.7.5
anyio               NA
attr                20.3.0
babel               2.9.0
backcall            0.2.0
brotli              NA
cairo               1.20.0
certifi             2020.12.05
cffi                1.14.5
chardet             4.0.0
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.0
dask                2021.03.1
dateutil            2.8.1
decorator           4.4.2
fsspec              0.8.7
get_version         2.1
google              NA
h5py                2.10.0
idna                2.10
igraph              0.8.3
ipykernel           5.5.0
ipython_genutils    0.2.0
jedi                0.18.0
jinja2              2.11.3
joblib              1.0.1
json5               NA
jsonschema          3.2.0
jupyter_server      1.4.1
jupyterlab_server   2.3.0
kiwisolver          1.3.1
legacy_api_wrap     0.0.0
leidenalg           0.8.3
llvmlite            0.34.0
louvain             0.7.0
markupsafe          1.1.1
mpl_toolkits        NA
natsort             7.1.1
nbclassic           NA
nbformat            5.1.2
numba               0.51.2
numexpr             2.7.3
packaging           20.9
parso               0.8.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
prometheus_client   NA
prompt_toolkit      3.0.16
psutil              5.8.0
ptyprocess          0.7.0
pvectorc            NA
pyarrow             0.16.0
pygments            2.8.0
pyparsing           2.4.7
pyrsistent          NA
pytoml              NA
pytz                2021.1
requests            2.25.1
ruamel              NA
scipy               1.6.1
send2trash          NA
setuptools_scm      NA
simplejson          3.17.2
sinfo               0.3.1
six                 1.15.0
sklearn             0.22
sniffio             1.2.0
socks               1.7.1
sphinxcontrib       NA
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
tlz                 0.11.0
toolz               0.11.1
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
uritemplate         3.0.1
urllib3             1.26.3
wcwidth             0.2.5
yaml                5.1.2
zmq                 22.0.3
```

Example 5 (unknown):
```unknown
-----
IPython             7.21.0
jupyter_client      6.1.11
jupyter_core        4.7.1
jupyterlab          3.0.9
notebook            6.2.0
-----
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0]
Linux-4.15.0-112-generic-x86_64-with-glibc2.10
-----
Session information updated at 2023-03-14 17:33
```

Example 6 (python):
```python
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/QC/T21_multiome_RNA_foetal_prefilter.h5ad')
adata
```

Example 7 (unknown):
```unknown
AnnData object with n_obs × n_vars = 190750 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-2', 'ambient_expression-3', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7'
    obsm: 'latent_gene_encoding'
```

Example 8 (python):
```python
adata.obs['combinedID'].value_counts()
```

Example 9 (unknown):
```unknown
BHF_F_Hea13188319_BHF_F_Hea13187621    50000
BHF_F_Hea13188321_BHF_F_Hea13187623    38944
BHF_F_Hea13188325_BHF_F_Hea13187627    23862
BHF_F_Hea13188322_BHF_F_Hea13187624    17452
BHF_F_Hea13188323_BHF_F_Hea13187625    17143
BHF_F_Hea13188327_BHF_F_Hea13187629    16377
BHF_F_Hea13188320_BHF_F_Hea13187622    14000
BHF_F_Hea13188326_BHF_F_Hea13187628    12972
Name: combinedID, dtype: int64
```

Example 10 (python):
```python
adata.obs['scrublet_score'].hist(bins=100)
```

Example 11 (unknown):
```unknown
<AxesSubplot:>
```

Example 12 (python):
```python
# set threshold (arbitrary)
dbl_score_thresh=0.15

prop = np.sum(adata.obs['scrublet_score']>=dbl_score_thresh)/len(adata)
print(f'proportion of doublets: {prop}')
```

Example 13 (unknown):
```unknown
proportion of doublets: 0.2766657929226737
```

Example 14 (python):
```python
# Save doublet ids
doublets = adata.obs_names[(adata.obs['scrublet_score']<dbl_score_thresh)==False]
with open('/nfs/team205/heart/anndata_objects/Foetal/QC/T21_Multiome_rm-doublets.txt', 'w') as f:
    for line in doublets:
        f.write(line)
        f.write('\n')

# Remove doublets
adata = adata[adata.obs['scrublet_score']<dbl_score_thresh]
print(adata.shape)
```

Example 15 (unknown):
```unknown
(137976, 36601)
```

Example 16 (python):
```python
sc.pp.filter_cells(adata, min_genes=500, inplace=True)
sc.pp.filter_cells(adata, min_counts=1000, inplace=True)
adata.shape
```

Example 17 (unknown):
```unknown
filtered out 19359 cells that have less than 500 genes expressed
Trying to set attribute `.obs` of view, copying.
filtered out 7603 cells that have less than 1000 counts
```

Example 18 (unknown):
```unknown
(111014, 36601)
```

Example 19 (python):
```python
adata.var['mt'] = adata.var['gene_name'].str.startswith('MT-')  # annotate the group of mitochondrial genes as 'mt'
adata.var['ribo'] = adata.var['gene_name'].str.startswith('RPS','RPL')  # annotate the group of ribosomal genes as 'ribo'
sc.pp.calculate_qc_metrics(adata, qc_vars=['mt','ribo'], percent_top=None, log1p=True, inplace=True)
```

Example 20 (python):
```python
plt.rcParams['figure.figsize'] = [3, 2]

sc.pl.scatter(adata, x='total_counts', y='n_genes')
sc.pl.scatter(adata, x='total_counts', y='pct_counts_mt')
sc.pl.scatter(adata, x='total_counts', y='pct_counts_ribo')
```

Example 21 (python):
```python
plt.rcParams['figure.figsize'] = [2, 2]
adata = adata[adata.obs['pct_counts_mt'] < 5, :]
adata = adata[adata.obs['pct_counts_ribo'] < 5, :]
sc.pl.violin(adata, ['n_genes', 'total_counts', 'log1p_total_counts','pct_counts_mt', 'pct_counts_ribo'],jitter = 0, multi_panel = True)
```

Example 22 (python):
```python
sc.settings.set_figure_params(dpi = 120, color_map = 'RdPu')
plt.rcParams['figure.figsize'] = 10,3
sc.pl.violin(adata,keys=['log1p_total_counts'],groupby='sangerID',rotation=90,jitter=False)
```

Example 23 (unknown):
```unknown
Trying to set attribute `.uns` of view, copying.
```

Example 24 (python):
```python
plt.rcParams['figure.figsize'] = 10,3
sc.pl.violin(adata,keys=['pct_counts_mt'],groupby='sangerID',rotation=90,jitter=False)
```

Example 25 (python):
```python
plt.rcParams['figure.figsize'] = 10,3
sc.pl.violin(adata,keys=['pct_counts_ribo'],groupby='sangerID',rotation=90,jitter=False)
```

Example 26 (python):
```python
sc.settings.set_figure_params(dpi = 120, color_map = 'RdPu')
```

Example 27 (python):
```python
adata.shape
```

Example 28 (unknown):
```unknown
(84159, 36601)
```

Example 29 (python):
```python
adata_c = adata.copy()

adata_c.var.set_index('gene_name',inplace=True)
adata_c.var_names = adata_c.var_names.astype('str')
adata_c.var_names_make_unique()

sc.pp.filter_genes(adata_c, min_cells=3)
sc.pp.normalize_total(adata_c, target_sum=1e4)
sc.pp.log1p(adata_c)
```

Example 30 (unknown):
```unknown
filtered out 4196 genes that are detected in less than 3 cells
normalizing counts per cell
    finished (0:00:00)
```

Example 31 (python):
```python
# scoring hemoglobin genes
# HGNC, Hemoglobin subunits, groupID:940
genes=['HBA1','HBA2','HBB','HBD','HBE1','HBG1','HBG2','HBM','HBQ1','HBZ']
sc.tl.score_genes(adata_c,gene_list=genes,score_name='HB_score')
```

Example 32 (python):
```python
computing score 'HB_score'
    finished: added
    'HB_score', score of gene set (adata.obs).
    400 total control genes are used. (0:00:03)
```

Example 33 (python):
```python
len(adata_c.obs['sangerID'].unique())
```

Example 34 (python):
```python
sc.settings.set_figure_params(dpi = 80, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')

fig,axs=plt.subplots(nrows=2,ncols=5,figsize=(15,5))

for i,ID in enumerate(adata_c.obs['sangerID'].unique()):
    sample = adata_c[adata_c.obs['sangerID']==ID]
    axs[i//5,i%5].hist(sample.obs['HB_score'],bins=100)
    axs[i//5,i%5].set_title(ID)

fig.tight_layout()
plt.show()
```

Example 35 (python):
```python
# add HB score to original adata
adata.obs.loc[adata_c.obs_names,'HB_score']=adata_c.obs['HB_score']
```

Example 36 (python):
```python
HBscore_thresh=1

adata=adata[adata.obs['HB_score']<HBscore_thresh]
print(adata.shape)
```

Example 37 (unknown):
```unknown
(82277, 36601)
```

Example 38 (python):
```python
adata.obs['combinedID'].value_counts()
```

Example 39 (unknown):
```unknown
BHF_F_Hea13188319_BHF_F_Hea13187621    21451
BHF_F_Hea13188321_BHF_F_Hea13187623    16869
BHF_F_Hea13188325_BHF_F_Hea13187627     9103
BHF_F_Hea13188323_BHF_F_Hea13187625     8115
BHF_F_Hea13188327_BHF_F_Hea13187629     7705
BHF_F_Hea13188322_BHF_F_Hea13187624     6868
BHF_F_Hea13188320_BHF_F_Hea13187622     6450
BHF_F_Hea13188326_BHF_F_Hea13187628     5716
Name: combinedID, dtype: int64
```

Example 40 (python):
```python
# save
adata.write('/nfs/team205/heart/anndata_objects/Foetal/QC/T21_multiome_RNA_foetal.h5ad')
adata
```

Example 41 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 82277 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-2', 'ambient_expression-3', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7', 'mt', 'ribo', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
    uns: 'sangerID_colors'
    obsm: 'latent_gene_encoding'
```

Example 42 (python):
```python
# read in 
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/QC/T21_multiome_RNA_foetal.h5ad')
adata
```

Example 43 (unknown):
```unknown
AnnData object with n_obs × n_vars = 82277 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-2', 'ambient_expression-3', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7', 'mt', 'ribo', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
    uns: 'sangerID_colors'
    obsm: 'latent_gene_encoding'
```

Example 44 (python):
```python
# set batch key and save again
adata.obs['batch_key']=adata.obs['donor'].astype('str')+'_'+adata.obs['cell_or_nuclei'].astype('str')+'_'+adata.obs['kit_10x'].astype('str')

# save
adata.write('/nfs/team205/heart/anndata_objects/Foetal/QC/T21_multiome_RNA_foetal.h5ad')

adata.obs['batch_key'].value_counts()
```

Example 45 (unknown):
```unknown
... storing 'batch_key' as categorical
```

Example 46 (unknown):
```unknown
Hst42_Nuclei_Multiome-v1    59753
Hst44_Nuclei_Multiome-v1    22524
Name: batch_key, dtype: int64
```

Example 47 (unknown):
```unknown
# rclone to gdrive
rclone copy /nfs/team205/heart/anndata_objects/Foetal/QC/T21_multiome_RNA_foetal.h5ad \
gdrive:JamesC_Kazumasa/Foetal_Heart/SanjayShinha/toSinhalab/anndata/
```

Example 48 (python):
```python
adata.obs['combinedID'].value_counts()
```

Example 49 (python):
```python
adata.obs['donor'].value_counts()
```

Example 50 (python):
```python
pd.crosstab(adata.obs['donor'],adata.obs['age'])
```

Example 51 (python):
```python
pd.crosstab(adata.obs['donor'],adata.obs['gender'])
```

Example 52 (python):
```python
adata.obs['SOC | status'].value_counts()
```

Example 53 (python):
```python
adata.obs[adata.obs['multiplexed']=='Yes']['donor_by_library-prep'].value_counts()
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/QC/Foetal_scRNA_include-doublets.html

**Contents:**
- Import modules¶
- Functions¶
- Read in anndata¶
- Remove low quality cells¶
- Compute QC stats¶
- ~Remove cells exceeding the ribo/mito threshold~¶
- Calculate RBC score¶
- Remove RBCs¶
- Save¶

**Examples:**

Example 1 (python):
```python
import numpy as np
import pandas as pd
import scanpy as sc
import matplotlib.pyplot as plt
import scrublet as scr
import session_info
```

Example 2 (python):
```python
sc.settings.verbosity = 3
sc.settings.set_figure_params(dpi = 120, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.3.4
numpy               1.20.1
pandas              1.1.4
scanpy              1.7.1
scrublet            NA
session_info        1.0.0
-----
```

Example 4 (unknown):
```unknown
PIL                 8.1.2
anndata             0.7.5
anyio               NA
attr                20.3.0
babel               2.9.0
backcall            0.2.0
brotli              NA
cairo               1.20.0
certifi             2020.12.05
cffi                1.14.5
chardet             4.0.0
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.0
dask                2021.03.1
dateutil            2.8.1
decorator           4.4.2
fsspec              0.8.7
get_version         2.1
google              NA
h5py                2.10.0
idna                2.10
igraph              0.8.3
ipykernel           5.5.0
ipython_genutils    0.2.0
jedi                0.18.0
jinja2              2.11.3
joblib              1.0.1
json5               NA
jsonschema          3.2.0
jupyter_server      1.4.1
jupyterlab_server   2.3.0
kiwisolver          1.3.1
legacy_api_wrap     0.0.0
leidenalg           0.8.3
llvmlite            0.34.0
louvain             0.7.0
markupsafe          1.1.1
mpl_toolkits        NA
natsort             7.1.1
nbclassic           NA
nbformat            5.1.2
numba               0.51.2
numexpr             2.7.3
packaging           20.9
parso               0.8.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
prometheus_client   NA
prompt_toolkit      3.0.16
psutil              5.8.0
ptyprocess          0.7.0
pvectorc            NA
pyarrow             0.16.0
pygments            2.8.0
pyparsing           2.4.7
pyrsistent          NA
pytoml              NA
pytz                2021.1
requests            2.25.1
ruamel              NA
scipy               1.6.1
send2trash          NA
setuptools_scm      NA
simplejson          3.17.2
sinfo               0.3.1
six                 1.15.0
sklearn             0.22
sniffio             1.2.0
socks               1.7.1
sphinxcontrib       NA
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
tlz                 0.11.0
toolz               0.11.1
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
uritemplate         3.0.1
urllib3             1.26.3
wcwidth             0.2.5
yaml                5.1.2
zmq                 22.0.3
```

Example 5 (unknown):
```unknown
-----
IPython             7.21.0
jupyter_client      6.1.11
jupyter_core        4.7.1
jupyterlab          3.0.9
notebook            6.2.0
-----
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0]
Linux-4.15.0-213-generic-x86_64-with-glibc2.10
-----
Session information updated at 2023-12-12 15:07
```

Example 6 (python):
```python
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/QC/scRNA_foetal_prefilter.h5ad')
adata
```

Example 7 (unknown):
```unknown
AnnData object with n_obs × n_vars = 205457 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-10', 'ambient_expression-11', 'ambient_expression-12', 'ambient_expression-13', 'ambient_expression-14', 'ambient_expression-15', 'ambient_expression-16', 'ambient_expression-17', 'ambient_expression-18', 'ambient_expression-19', 'ambient_expression-2', 'ambient_expression-3', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7', 'ambient_expression-8', 'ambient_expression-9'
    obsm: 'latent_gene_encoding'
```

Example 8 (python):
```python
adata.obs['sangerID'].value_counts()
```

Example 9 (unknown):
```unknown
Sinha_F2             24275
Sinha_E2             24227
BHF_F_Hea11192328    21714
Sinha_E8             17219
BHF_F_Hea11192324    16688
BHF_F_Hea11192326    14117
BHF_F_Hea11192325    10676
Sinha_F4             10652
BHF_F_Hea11192327     9130
BHF_F_Hea10402918     8010
Sinha_F7              6973
Sinha_D8              6714
Sinha_A1              6640
Sinha_B1              5988
BHF_F_Hea11192323     4893
Sinha_C8              4587
BHF_F_Hea10402917     3752
Sinha_G4              3389
Sinha_F8              3203
Sinha_G7              2610
Name: sangerID, dtype: int64
```

Example 10 (python):
```python
sc.pp.filter_cells(adata, min_genes=500, inplace=True)
sc.pp.filter_cells(adata, min_counts=1000, inplace=True)
print(adata.shape)
```

Example 11 (unknown):
```unknown
filtered out 62633 cells that have less than 500 genes expressed
filtered out 2190 cells that have less than 1000 counts
```

Example 12 (unknown):
```unknown
(140634, 36601)
```

Example 13 (python):
```python
adata.obs['sangerID'].value_counts()
```

Example 14 (unknown):
```unknown
BHF_F_Hea11192328    15321
BHF_F_Hea11192326    13251
BHF_F_Hea11192324    13229
Sinha_E2             11525
Sinha_E8             10036
BHF_F_Hea11192325     9421
Sinha_F2              9259
Sinha_F4              9215
BHF_F_Hea11192327     7387
Sinha_D8              6107
BHF_F_Hea10402918     5734
Sinha_F7              5547
Sinha_A1              4671
BHF_F_Hea11192323     4000
Sinha_C8              3981
Sinha_B1              3606
Sinha_G4              2604
BHF_F_Hea10402917     2517
Sinha_G7              1779
Sinha_F8              1444
Name: sangerID, dtype: int64
```

Example 15 (python):
```python
adata.var['mt'] = adata.var['gene_name'].str.startswith('MT-')  # annotate the group of mitochondrial genes as 'mt'
adata.var['ribo'] = adata.var['gene_name'].str.startswith('RPS','RPL')  # annotate the group of ribosomal genes as 'mt'
sc.pp.calculate_qc_metrics(adata, qc_vars=['mt','ribo'], percent_top=None, log1p=True, inplace=True)
```

Example 16 (python):
```python
plt.rcParams['figure.figsize'] = [3, 2]

print('### Sanger ###')
fig,axs=plt.subplots(nrows=1,ncols=3,figsize=(10,3))
sc.pl.scatter(adata[adata.obs['facility']=='Sanger'], x='total_counts', y='n_genes', show=False, ax=axs[0])
sc.pl.scatter(adata[adata.obs['facility']=='Sanger'], x='total_counts', y='pct_counts_mt', show=False, ax=axs[1])
sc.pl.scatter(adata[adata.obs['facility']=='Sanger'], x='total_counts', y='pct_counts_ribo', show=False, ax=axs[2])

fig.tight_layout()
plt.show()

plt.rcParams['figure.figsize'] = [3, 2]

print('### CSCI ###')
fig,axs=plt.subplots(nrows=1,ncols=3,figsize=(10,3))
sc.pl.scatter(adata[adata.obs['facility']=='CSCI'], x='total_counts', y='n_genes', show=False, ax=axs[0])
sc.pl.scatter(adata[adata.obs['facility']=='CSCI'], x='total_counts', y='pct_counts_mt', show=False, ax=axs[1])
sc.pl.scatter(adata[adata.obs['facility']=='CSCI'], x='total_counts', y='pct_counts_ribo', show=False, ax=axs[2])

fig.tight_layout()
plt.show()
```

Example 17 (unknown):
```unknown
### Sanger ###
```

Example 18 (unknown):
```unknown
### CSCI ###
```

Example 19 (python):
```python
'''
plt.rcParams['figure.figsize'] = [2, 2]
adata = adata[adata.obs['pct_counts_mt'] < 20, :]
adata = adata[adata.obs['pct_counts_ribo'] < 20, :]
sc.pl.violin(adata, ['n_genes', 'total_counts','log1p_total_counts', 'pct_counts_mt', 'pct_counts_ribo'],jitter = 0.2, multi_panel = True)
'''
```

Example 20 (python):
```python
'''
sc.settings.set_figure_params(dpi = 120, color_map = 'RdPu')
plt.rcParams['figure.figsize'] = [8, 2]
sc.pl.violin(adata,keys=['log1p_total_counts'],groupby='sangerID',rotation=90,jitter=False)
'''
```

Example 21 (python):
```python
# print(adata.shape)
# adata.obs['sangerID'].value_counts()
```

Example 22 (python):
```python
adata_c = adata.copy()

adata_c.var.set_index('gene_name',inplace=True)
adata_c.var_names = adata_c.var_names.astype('str')
adata_c.var_names_make_unique()

sc.pp.filter_genes(adata_c, min_cells=3)
sc.pp.normalize_total(adata_c, target_sum=1e4)
sc.pp.log1p(adata_c)
```

Example 23 (unknown):
```unknown
filtered out 5009 genes that are detected in less than 3 cells
normalizing counts per cell
    finished (0:00:01)
```

Example 24 (python):
```python
# scoring hemoglobin genes
# HGNC, Hemoglobin subunits, groupID:940
genes=['HBA1','HBA2','HBB','HBD','HBE1','HBG1','HBG2','HBM','HBQ1','HBZ']
sc.tl.score_genes(adata_c,gene_list=genes,score_name='HB_score')
```

Example 25 (python):
```python
computing score 'HB_score'
    finished: added
    'HB_score', score of gene set (adata.obs).
    250 total control genes are used. (0:00:05)
```

Example 26 (python):
```python
len(adata_c.obs['sangerID'].unique())
```

Example 27 (python):
```python
sc.settings.set_figure_params(dpi = 80, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')

fig,axs=plt.subplots(nrows=4,ncols=5,figsize=(15,7))

for i,ID in enumerate(adata_c.obs['sangerID'].unique()):
    sample = adata_c[adata_c.obs['sangerID']==ID]
    axs[i//5,i%5].hist(sample.obs['HB_score'],bins=100)
    axs[i//5,i%5].set_title(ID)

fig.tight_layout()
plt.show()
```

Example 28 (python):
```python
# add HB score to original adata
adata.obs.loc[adata_c.obs_names,'HB_score']=adata_c.obs['HB_score']
```

Example 29 (python):
```python
HBscore_thresh=2

adata=adata[adata.obs['HB_score']<HBscore_thresh]
print(adata.shape)
```

Example 30 (unknown):
```unknown
(121458, 36601)
```

Example 31 (python):
```python
# save
adata.write('/nfs/team205/heart/anndata_objects/Foetal/QC/scRNA_foetal_include-doublets_13Dec2023.h5ad')
adata
```

Example 32 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 121458 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-10', 'ambient_expression-11', 'ambient_expression-12', 'ambient_expression-13', 'ambient_expression-14', 'ambient_expression-15', 'ambient_expression-16', 'ambient_expression-17', 'ambient_expression-18', 'ambient_expression-19', 'ambient_expression-2', 'ambient_expression-3', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7', 'ambient_expression-8', 'ambient_expression-9', 'mt', 'ribo', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
    obsm: 'latent_gene_encoding'
```

Example 33 (python):
```python
adata.obs['sangerID'].value_counts()
```

Example 34 (unknown):
```unknown
BHF_F_Hea11192328    15280
BHF_F_Hea11192326    13229
BHF_F_Hea11192324    13207
BHF_F_Hea11192325     9418
Sinha_E8              8789
Sinha_F4              8655
BHF_F_Hea11192327     7381
BHF_F_Hea10402918     5675
Sinha_D8              5431
Sinha_F7              5157
Sinha_F2              4874
Sinha_E2              4377
BHF_F_Hea11192323     3998
Sinha_C8              3387
Sinha_A1              2921
BHF_F_Hea10402917     2510
Sinha_B1              2330
Sinha_G4              2250
Sinha_G7              1494
Sinha_F8              1095
Name: sangerID, dtype: int64
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/QC/Demultiplex/No2_CMO_download_results.html

**Contents:**
- Download¶
- Read in path file¶
- Download from iRODs¶
- Read in Multiome data¶
- CMO result¶
- Save¶
- Proportions¶

**Examples:**

Example 1 (python):
```python
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import numpy as np
import pandas as pd
import scanpy as sc
import session_info
import matplotlib.pyplot as plt

import os
```

Example 2 (unknown):
```unknown
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.5.2
numpy               1.21.5
pandas              1.3.5
scanpy              1.8.2
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                 9.1.1
anndata             0.8.0
asciitree           NA
backcall            0.2.0
beta_ufunc          NA
binom_ufunc         NA
cached_property     1.5.1
cairo               1.20.0
cffi                1.14.4
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.2
dask                2022.01.0
dateutil            2.8.1
decorator           4.4.2
defusedxml          0.6.0
fasteners           0.17.3
fsspec              2022.01.0
google              NA
h5py                3.1.0
igraph              0.10.2
importlib_metadata  1.7.0
ipykernel           5.4.2
ipython_genutils    0.2.0
jedi                0.17.2
jinja2              3.1.2
joblib              1.0.0
kiwisolver          1.3.1
leidenalg           0.9.0
llvmlite            0.35.0
louvain             0.7.0
markupsafe          2.1.1
mpl_toolkits        NA
msgpack             1.0.3
natsort             7.1.0
nbinom_ufunc        NA
numba               0.52.0
numcodecs           0.9.1
numexpr             2.7.2
packaging           21.3
parso               0.7.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
plotly              5.10.0
prompt_toolkit      3.0.10
psutil              5.9.0
ptyprocess          0.7.0
pyarrow             9.0.0
pygments            2.13.0
pyparsing           2.4.7
pytz                2020.5
scipy               1.7.3
setuptools_scm      NA
sinfo               0.3.1
six                 1.14.0
sklearn             1.0.2
sphinxcontrib       NA
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
threadpoolctl       2.1.0
tlz                 0.11.2
toolz               0.11.2
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
wcwidth             0.2.5
yaml                5.4.1
zarr                2.11.3
zipp                NA
zmq                 21.0.1
zope                NA
```

Example 5 (unknown):
```unknown
-----
IPython             7.19.0
jupyter_client      6.1.11
jupyter_core        4.7.0
notebook            6.2.0
-----
Python 3.7.9 | packaged by conda-forge | (default, Dec  9 2020, 21:08:20) [GCC 9.3.0]
Linux-4.15.0-112-generic-x86_64-with-debian-bullseye-sid
-----
Session information updated at 2023-02-07 23:34
```

Example 6 (python):
```python
samplefile = pd.read_csv('/nfs/team205/heart/mapped/CMO/csv/multiome-foetal_20221221.csv')
samplefile.set_index('CMO_ID',inplace=True)
samplefile
```

Example 7 (unknown):
```unknown
cd /nfs/team205/heart/mapped/CMO
sed 1d ./csv/multiome-foetal_20221221.csv | while IFS=, read -r mID cID IRODSPATH
do
echo $cID

# create output folder if it does not exist
if [[ ! -d "$cID" ]]; then
    mkdir -p "$cID"
fi

iget -K -r $IRODSPATH/raw_feature_bc_matrix $cID
iget -K $IRODSPATH/web_summary.html ./websummary/multiome-foetal_20221221/${cID}_web_summary.html
done
```

Example 8 (unknown):
```unknown
cID=7089STDY13216919
IRODSPATH=/seq/illumina/runs/46/46301/cellranger/cellranger700_count_46301_7089STDY13216919_GRCh38-2020-A
iget -K -r $IRODSPATH/raw_feature_bc_matrix $cID
iget -K $IRODSPATH/web_summary.html ./websummary/multiome-foetal_20221221/${cID}_web_summary.html
```

Example 9 (python):
```python
# post-cellbender
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/QC/multiome_RNA_foetal_prefilter.h5ad',
                     backed = 'r')
# only multiplexed data
adata = adata[adata.obs['combinedID'].isin(list(samplefile['Multiome_ID']))]

adata
```

Example 10 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 74984 × 36601 backed at '/nfs/team205/heart/anndata_objects/Foetal/QC/multiome_RNA_foetal_prefilter.h5ad'
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-10', 'ambient_expression-11', 'ambient_expression-12', 'ambient_expression-13', 'ambient_expression-14', 'ambient_expression-15', 'ambient_expression-16', 'ambient_expression-17', 'ambient_expression-18', 'ambient_expression-19', 'ambient_expression-2', 'ambient_expression-20', 'ambient_expression-21', 'ambient_expression-22', 'ambient_expression-23', 'ambient_expression-24', 'ambient_expression-25', 'ambient_expression-26', 'ambient_expression-27', 'ambient_expression-28', 'ambient_expression-29', 'ambient_expression-3', 'ambient_expression-30', 'ambient_expression-31', 'ambient_expression-32', 'ambient_expression-33', 'ambient_expression-34', 'ambient_expression-35', 'ambient_expression-36', 'ambient_expression-37', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7', 'ambient_expression-8', 'ambient_expression-9'
    obsm: 'latent_gene_encoding'
```

Example 11 (python):
```python
adata.obs['combinedID'].value_counts()
```

Example 12 (unknown):
```unknown
7089STDY13216920_BHF_F_Hea13242527    27498
7089STDY13216921_BHF_F_Hea13242528    24211
7089STDY13216922_BHF_F_Hea13242529    21402
7089STDY13216923_BHF_F_Hea13242530     1873
Name: combinedID, dtype: int64
```

Example 13 (unknown):
```unknown
* donor_assignment: Determining by selecting which of its donors (tags) has the maximum count per cell
* ratio: Ratio of "the 1st maximum tag count" to "the 2nd maximum tag count" per cell.
```

Example 14 (python):
```python
# function to calculate proportion of the maximum tag count to the total tag count

def ratio(row):
    # convert to array
    arr = np.array(row)
    # sort -> reverse to make it descending
    arr.sort()
    arr = arr[::-1]
    # get 1st and 2nd max value
    first = arr[0]
    second = arr[1]
    # return ratio
    return first/second
```

Example 15 (python):
```python
for i,cmo_ID in enumerate(samplefile.index):
    # read in
    cmo = sc.read_10x_mtx(f'/nfs/team205/heart/mapped/CMO/{cmo_ID}/raw_feature_bc_matrix',
                     gex_only=False)
    
    # get multiome ID
    multi_ID = samplefile.loc[cmo_ID,'Multiome_ID']

    # modify obs_names
    cmo.obs_names = multi_ID + '_' + cmo.obs_names
    # add sample IDs
    cmo.obs['combinedID'] = multi_ID
    
    # select cells in post-cellbender adata
    obsnames = adata.obs.index[adata.obs['combinedID']==multi_ID]
    cmo = cmo[obsnames]
    # get obs
    cmo_obs = cmo.obs.copy()
    
    # prepare count dataframe
    X_df = cmo.to_df()
    X_df.columns = [x.split('-')[0] for x in X_df.columns]
    
    # calculate delta
    cmo_obs['ratio'] = X_df.apply(ratio,axis=1)
    cmo_obs['top_count'] = X_df.max(axis=1)
    
    # assign donor
    def which_is_max(row):
        # convert to array
        arr = np.array(row)
        # get position
        pos = np.where(arr == np.max(arr))[0]
        # get donor
        don = ','.join(X_df.columns[pos])
        return don
    def n_max_donor(row):
        # convert to array
        arr = np.array(row)
        # get position
        pos = np.where(arr == np.max(arr))[0]
        return len(pos)
    cmo_obs['donor_assignment'] = X_df.apply(which_is_max,axis=1)
    cmo_obs['n_donor_assigned'] = X_df.apply(n_max_donor,axis=1)
    
    # add count values
    cmo_obs[X_df.columns] = X_df.copy()
    
    # concatenate
    if i==0:
        cmo_obs_all = cmo_obs.copy()
    else:
        cmo_obs_all = pd.concat([cmo_obs_all,cmo_obs])
        
    del cmo, cmo_obs, multi_ID, obsnames, X_df

print(cmo_obs_all.shape)
cmo_obs_all.head()
```

Example 16 (unknown):
```unknown
# check samples which is 1st max == 2d max
cmo_obs_all[cmo_obs_all['n_donor_assigned']==2]
```

Example 17 (unknown):
```unknown
cmo_obs_all.to_csv('/nfs/team205/heart/mapped/CMO/result_summary/CMO-result_multiome-foetal_20230207.csv')
```

Example 18 (python):
```python
df = cmo_obs_all.copy()
plt.rcParams['figure.figsize'] = 3,3
tmp = pd.crosstab(df['combinedID'], df['donor_assignment'], normalize='index')
# tmp.plot.bar().legend(loc=1)
ax = tmp.plot.bar(stacked=True,width=0.7)
ax.grid(False)
ax.set_ylabel('proportion')
ax.legend(bbox_to_anchor=(1.0, 1.0))
ax.plot()
plt.show()

for sam in list(samplefile['Multiome_ID']):
    df_s = df[df['combinedID']==sam]
    print(f'##### {sam} #####')
    print(df_s['donor_assignment'].value_counts())
    print('')
```

Example 19 (unknown):
```unknown
##### 7089STDY13216920_BHF_F_Hea13242527 #####
Hst40          18933
Hst39           8557
Hst40,Hst39        8
Name: donor_assignment, dtype: int64

##### 7089STDY13216921_BHF_F_Hea13242528 #####
Hst40          15866
Hst39           8335
Hst40,Hst39       10
Name: donor_assignment, dtype: int64

##### 7089STDY13216922_BHF_F_Hea13242529 #####
Hst40          13755
Hst36           5466
Hst39           2178
Hst40,Hst36        2
Hst40,Hst39        1
Name: donor_assignment, dtype: int64

##### 7089STDY13216923_BHF_F_Hea13242530 #####
Hst40    1435
Hst36     394
Hst39      44
Name: donor_assignment, dtype: int64
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/QC/Foetal_scRNA_No1_make-anndata_run-scrublet.html

**Contents:**
- Import modules¶
- Functions¶
- Create anndata object¶
- Run scrublet¶
- Save¶

205457 rows × 21 columns

**Examples:**

Example 1 (python):
```python
import numpy as np
import pandas as pd
import scanpy as sc
import matplotlib.pyplot as plt
import scrublet as scr
import session_info
```

Example 2 (python):
```python
sc.settings.verbosity = 3
sc.settings.set_figure_params(dpi = 120, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.3.4
numpy               1.20.1
pandas              1.1.4
scanpy              1.7.1
scrublet            NA
session_info        1.0.0
-----
```

Example 4 (unknown):
```unknown
PIL                 8.1.2
anndata             0.7.5
anyio               NA
attr                20.3.0
babel               2.9.0
backcall            0.2.0
brotli              NA
cairo               1.20.0
certifi             2020.12.05
cffi                1.14.5
chardet             4.0.0
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.0
dask                2021.03.1
dateutil            2.8.1
decorator           4.4.2
fsspec              0.8.7
get_version         2.1
google              NA
h5py                2.10.0
idna                2.10
igraph              0.8.3
ipykernel           5.5.0
ipython_genutils    0.2.0
jedi                0.18.0
jinja2              2.11.3
joblib              1.0.1
json5               NA
jsonschema          3.2.0
jupyter_server      1.4.1
jupyterlab_server   2.3.0
kiwisolver          1.3.1
legacy_api_wrap     0.0.0
leidenalg           0.8.3
llvmlite            0.34.0
louvain             0.7.0
markupsafe          1.1.1
mpl_toolkits        NA
natsort             7.1.1
nbclassic           NA
nbformat            5.1.2
numba               0.51.2
numexpr             2.7.3
packaging           20.9
parso               0.8.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
prometheus_client   NA
prompt_toolkit      3.0.16
psutil              5.8.0
ptyprocess          0.7.0
pvectorc            NA
pyarrow             0.16.0
pygments            2.8.0
pyparsing           2.4.7
pyrsistent          NA
pytoml              NA
pytz                2021.1
requests            2.25.1
ruamel              NA
scipy               1.6.1
send2trash          NA
setuptools_scm      NA
simplejson          3.17.2
sinfo               0.3.1
six                 1.15.0
sklearn             0.22
sniffio             1.2.0
socks               1.7.1
sphinxcontrib       NA
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
tlz                 0.11.0
toolz               0.11.1
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
uritemplate         3.0.1
urllib3             1.26.3
wcwidth             0.2.5
yaml                5.1.2
zmq                 22.0.3
```

Example 5 (unknown):
```unknown
-----
IPython             7.21.0
jupyter_client      6.1.11
jupyter_core        4.7.1
jupyterlab          3.0.9
notebook            6.2.0
-----
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0]
Linux-4.15.0-158-generic-x86_64-with-glibc2.10
-----
Session information updated at 2022-12-23 09:18
```

Example 6 (python):
```python
"""Functions for downstream work with outputs of remove-background."""

# to read cellbender outputs<br>
# https://github.com/broadinstitute/CellBender/issues/57

import tables
import numpy as np
import scipy.sparse as sp
from typing import Dict


def dict_from_h5(file: str) -> Dict[str, np.ndarray]:
    """Read in everything from an h5 file and put into a dictionary."""
    d = {}
    with tables.open_file(file) as f:
        # read in everything
        for array in f.walk_nodes("/", "Array"):
            d[array.name] = array.read()
    return d


def anndata_from_h5(file: str,
                    analyzed_barcodes_only: bool = True) -> 'anndata.AnnData':
    """Load an output h5 file into an AnnData object for downstream work.

    Args:
        file: The h5 file
        analyzed_barcodes_only: False to load all barcodes, so that the size of
            the AnnData object will match the size of the input raw count matrix.
            True to load a limited set of barcodes: only those analyzed by the
            algorithm. This allows relevant latent variables to be loaded
            properly into adata.obs and adata.obsm, rather than adata.uns.

    Returns:
        adata: The anndata object, populated with inferred latent variables
            and metadata.

    """

    try:
        import anndata
    except ImportError:
        raise ImportError('The anndata package must be installed to use the '
                          'function anndata_from_h5()')

    d = dict_from_h5(file)
    X = sp.csc_matrix((d.pop('data'), d.pop('indices'), d.pop('indptr')),
                      shape=d.pop('shape')).transpose().tocsr()

    if analyzed_barcodes_only:
        if 'barcodes_analyzed_inds' in d.keys():
            X = X[d['barcodes_analyzed_inds'], :]
            d['barcodes'] = d['barcodes'][d['barcodes_analyzed_inds']]
        elif 'barcode_indices_for_latents' in d.keys():
            X = X[d['barcode_indices_for_latents'], :]
            d['barcodes'] = d['barcodes'][d['barcode_indices_for_latents']]
        else:
            print('Warning: analyzed_barcodes_only=True, but the key '
                  '"barcodes_analyzed_inds" or "barcode_indices_for_latents" '
                  'is missing from the h5 file. '
                  'Will output all barcodes, and proceed as if '
                  'analyzed_barcodes_only=False')

    # Construct the count matrix.
    adata = anndata.AnnData(X=X,
                            obs={'barcode': d.pop('barcodes').astype(str)},
                            var={'gene_name': (d.pop('gene_names') if 'gene_names' in d.keys()
                                               else d.pop('name')).astype(str)})
    adata.obs.set_index('barcode', inplace=True)
    adata.var.set_index('gene_name', inplace=True)

    # Add other information to the adata object in the appropriate slot.
    for key, value in d.items():
        try:
            value = np.asarray(value)
            if len(value.shape) == 0:
                adata.uns[key] = value
            elif value.shape[0] == X.shape[0]:
                if (len(value.shape) < 2) or (value.shape[1] < 2):
                    adata.obs[key] = value
                else:
                    adata.obsm[key] = value
            elif value.shape[0] == X.shape[1]:
                if value.dtype.name.startswith('bytes'):
                    adata.var[key] = value.astype(str)
                else:
                    adata.var[key] = value
            else:
                adata.uns[key] = value
        except Exception:
            print('Unable to load data into AnnData: ', key, value, type(value))

    if analyzed_barcodes_only:
        for col in adata.obs.columns[adata.obs.columns.str.startswith('barcodes_analyzed')
                                     | adata.obs.columns.str.startswith('barcode_indices')]:
            try:
                del adata.obs[col]
            except Exception:
                pass

    return adata
```

Example 7 (python):
```python
# from https://github.com/Teichlab/mapcloud/blob/master/scripts/starsolo/postprocess.py

from statsmodels.stats.multitest import multipletests
# from emptydrops.matrix import CountMatrix
# from emptydrops import find_nonambient_barcodes
import scrublet as scr
import scanpy as sc
import pandas as pd
import numpy as np
import scipy

#some functions that Ni uses in scanpy scripts to run scrublet
#which in turn are inspired by my original notebook on the matter
#(extracted from scanpy_scripts 0.2.10 to get around scanpy version incompatibility)
def test_outlier(x, upper_mad_only=True):
	med = np.median(x)
	if upper_mad_only:
		mad = np.median(x[x>med] - med) * 1.4826
	else:
		mad = np.median(np.abs(x - med)) * 1.4826
	pvals = 1 - scipy.stats.norm.cdf(x, loc=med, scale=mad)
	bh_pvals = multipletests(pvals, method='fdr_bh')[1]
	return pvals, bh_pvals

def run_scrublet(adata, resolution_function=None):
	old_verbosity = sc.settings.verbosity
	sc.settings.verbosity = 1
	if resolution_function is None:
		resolution_function = lambda x: np.maximum(np.maximum(np.log10(x)-1, 0)**2, 0.1)
	scrub = scr.Scrublet(adata.X)
	#this has the potential to brick for poor quality data
	#if so, abort it and everything downstream
	try:
		ds, pd = scrub.scrub_doublets(verbose=False)
	except:
		return
	adata.obs['scrublet_score'] = ds

	adata_copy = adata.copy()
	sc.pp.filter_genes(adata_copy, min_cells=3)
	sc.pp.normalize_total(adata_copy, target_sum=1e4)
	sc.pp.log1p(adata_copy)
	sc.pp.highly_variable_genes(adata_copy, min_mean=0.0125, max_mean=3, min_disp=0.5, subset=True)
	sc.pp.scale(adata_copy, zero_center=False)
	sc.pp.pca(adata_copy, svd_solver='arpack', zero_center=False)
	sc.pp.neighbors(adata_copy, n_pcs=30)
	sc.tl.umap(adata_copy)
	sc.tl.leiden(adata_copy, resolution=1)
	for clst in np.unique(adata_copy.obs['leiden']):
		clst_size = sum(adata_copy.obs['leiden'] == clst)
		sc.tl.leiden(adata_copy, restrict_to=('leiden', [clst]), resolution=resolution_function(clst_size), key_added='leiden_R')
		adata_copy.obs['leiden'] = adata_copy.obs['leiden_R']
	clst_meds = []
	for clst in np.unique(adata_copy.obs['leiden']):
		k = adata_copy.obs['leiden'] == clst
		clst_med = np.median(adata_copy.obs.loc[k, 'scrublet_score'])
		adata_copy.obs.loc[k, 'cluster_scrublet_score'] = clst_med
		clst_meds.append(clst_med)
	clst_meds = np.array(clst_meds)
	pvals, bh_pvals = test_outlier(clst_meds)
	for i, clst in enumerate(np.unique(adata_copy.obs['leiden'])):
		k = adata_copy.obs['leiden'] == clst
		adata_copy.obs.loc[k, 'pval'] = pvals[i]
		adata_copy.obs.loc[k, 'bh_pval'] = bh_pvals[i]
	sc.settings.verbosity = old_verbosity
	#need to also export the clustering, for soupx purposes
	adata.obs['scrublet_leiden'] = adata_copy.obs['leiden']
	adata.obs['scrublet_score'] = adata_copy.obs['scrublet_score']
	adata.obs['cluster_scrublet_score'] = adata_copy.obs['cluster_scrublet_score']
	adata.obs['doublet_pval'] = adata_copy.obs['pval']
	adata.obs['doublet_bh_pval'] = adata_copy.obs['bh_pval']
	del adata_copy
```

Example 8 (python):
```python
# read in metadata
metadata = pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20221223.csv', sep = ',', index_col = None)
metadata = metadata[metadata['Publication']=='Foetal']

# select healthy data
metadata = metadata[metadata['status']=='Healthy']


print(metadata['modality'].value_counts())
```

Example 9 (unknown):
```unknown
Multiome-ATAC    38
Multiome-RNA     38
Visium           23
scRNA            20
Name: modality, dtype: int64
```

Example 10 (python):
```python
metadata['region'].value_counts()
```

Example 11 (unknown):
```unknown
Heart                                                            59
Apex                                                              8
Left ventricle                                                    7
Base                                                              6
Outflow Tract                                                     5
Pulmonary arches and branches                                     4
Aorta (Hst39), Nodes (Hst40)                                      4
Aorta                                                             4
Pericardium                                                       4
Heart-wo-nodes                                                    4
Heart (Hst36), Heart-wo-nodes (Hst39), Heart-wo-nodes (Hst40)     4
Ductus Arterious                                                  4
Atrial septum                                                     2
Right ventricle + Papillary                                       2
Nodes                                                             2
Name: region, dtype: int64
```

Example 12 (python):
```python
# select modality
metadata = metadata[metadata['modality']=='scRNA']
```

Example 13 (python):
```python
pd.crosstab(metadata['kit_10x'], metadata['facility'])
```

Example 14 (python):
```python
########### editted: put id to var_names ###########

# read in
adatas=[]
for i in range(len(metadata)):
    print(metadata.iloc[i]['sangerID'])
    
    path=metadata.iloc[i]['CellBender_out']
    adatas.append(anndata_from_h5(path + '/' + path.split('/')[-1] + '_cellbender_out_filtered.h5', analyzed_barcodes_only=False))
    del path
    
    # replace var_names with ensembleID
    adatas[i].var.reset_index(inplace=True)
    if 'id' in adatas[i].var.columns:
        adatas[i].var.set_index('id',inplace=True)
    else:
        adatas[i].var.set_index('genes',inplace=True)
    
    # modify barcodes
    adatas[i].obs.index = metadata.iloc[i]['sangerID'] + '_' + adatas[i].obs.index 
    
    # add metadata
    for col in ['sangerID','combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age',
                'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x','flushed']:
        adatas[i].obs[col] = metadata.iloc[i][col]


# concatenate
adata = adatas[0].concatenate(adatas[1:], index_unique = None, batch_key=None)
adata.shape
```

Example 15 (unknown):
```unknown
BHF_F_Hea10402917
BHF_F_Hea10402918
BHF_F_Hea11192323
BHF_F_Hea11192324
BHF_F_Hea11192325
BHF_F_Hea11192326
BHF_F_Hea11192327
BHF_F_Hea11192328
Sinha_A1
Sinha_B1
Sinha_C8
Sinha_D8
Sinha_E2
Sinha_E8
Sinha_F2
Sinha_F4
Sinha_F7
Sinha_F8
Sinha_G4
Sinha_G7
```

Example 16 (unknown):
```unknown
(205457, 36601)
```

Example 17 (python):
```python
for i in range(len(adatas)):
    print(adatas[i].var.columns)
```

Example 18 (unknown):
```unknown
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
```

Example 19 (python):
```python
adata.obs['sangerID'].value_counts()
```

Example 20 (unknown):
```unknown
Sinha_F2             24275
Sinha_E2             24227
BHF_F_Hea11192328    21714
Sinha_E8             17219
BHF_F_Hea11192324    16688
BHF_F_Hea11192326    14117
BHF_F_Hea11192325    10676
Sinha_F4             10652
BHF_F_Hea11192327     9130
BHF_F_Hea10402918     8010
Sinha_F7              6973
Sinha_D8              6714
Sinha_A1              6640
Sinha_B1              5988
BHF_F_Hea11192323     4893
Sinha_C8              4587
BHF_F_Hea10402917     3752
Sinha_G4              3389
Sinha_F8              3203
Sinha_G7              2610
Name: sangerID, dtype: int64
```

Example 21 (python):
```python
adata.obs['sangerID'].unique()
```

Example 22 (unknown):
```unknown
array(['BHF_F_Hea10402917', 'BHF_F_Hea10402918', 'BHF_F_Hea11192323',
       'BHF_F_Hea11192324', 'BHF_F_Hea11192325', 'BHF_F_Hea11192326',
       'BHF_F_Hea11192327', 'BHF_F_Hea11192328', 'Sinha_A1', 'Sinha_B1',
       'Sinha_C8', 'Sinha_D8', 'Sinha_E2', 'Sinha_E8', 'Sinha_F2',
       'Sinha_F4', 'Sinha_F7', 'Sinha_F8', 'Sinha_G4', 'Sinha_G7'],
      dtype=object)
```

Example 23 (python):
```python
%%time

# per rxn
for i,ID in enumerate(adata.obs['sangerID'].unique()):
    print(ID)
    
    ad = adata[adata.obs['sangerID'] == ID].copy()
    run_scrublet(ad)
    if i==0:
        meta = ad.obs
    else:
        meta = pd.concat([meta, ad.obs])
    del ad
```

Example 24 (unknown):
```unknown
BHF_F_Hea10402917
```

Example 25 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 26 (unknown):
```unknown
BHF_F_Hea10402918
```

Example 27 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 28 (unknown):
```unknown
BHF_F_Hea11192323
```

Example 29 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 30 (unknown):
```unknown
BHF_F_Hea11192324
```

Example 31 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 32 (unknown):
```unknown
BHF_F_Hea11192325
```

Example 33 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 34 (unknown):
```unknown
BHF_F_Hea11192326
```

Example 35 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 36 (unknown):
```unknown
BHF_F_Hea11192327
```

Example 37 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 38 (unknown):
```unknown
BHF_F_Hea11192328
```

Example 39 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 40 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 41 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 42 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 43 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 44 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 45 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 46 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 47 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 48 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 49 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 50 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 51 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 52 (unknown):
```unknown
CPU times: user 57min 17s, sys: 50min 50s, total: 1h 48min 7s
Wall time: 24min 42s
```

Example 53 (python):
```python
meta_scrub = meta[['scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval']].copy()
meta_scrub.shape
```

Example 54 (unknown):
```unknown
(205457, 5)
```

Example 55 (python):
```python
if meta_scrub.reindex(adata.obs.index).index.equals(adata.obs.index):
    adata.obs = pd.concat([adata.obs, meta_scrub.reindex(adata.obs.index)], axis=1)
else:
    raise Exception('Different barcodes in meta and adata')
```

Example 56 (python):
```python
adata.write('/nfs/team205/heart/anndata_objects/Foetal/QC/scRNA_foetal_prefilter.h5ad')
```

Example 57 (unknown):
```unknown
... storing 'sangerID' as categorical
... storing 'combinedID' as categorical
... storing 'donor' as categorical
... storing 'donor_type' as categorical
... storing 'region' as categorical
... storing 'region_finest' as categorical
... storing 'age' as categorical
... storing 'gender' as categorical
... storing 'facility' as categorical
... storing 'cell_or_nuclei' as categorical
... storing 'modality' as categorical
... storing 'kit_10x' as categorical
... storing 'flushed' as categorical
... storing 'scrublet_leiden' as categorical
... storing 'gene_name' as categorical
... storing 'feature_type' as categorical
```

Example 58 (python):
```python
sc.pl.violin(adata[adata.obs['facility']=='Sanger'],
             ['scrublet_score', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval'],jitter = 0.2, multi_panel = True)

sc.pl.violin(adata[adata.obs['facility']=='CSCI'],
             ['scrublet_score', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval'],jitter = 0.2, multi_panel = True)
```

Example 59 (python):
```python
adata.obs['sangerID'].value_counts()
```

Example 60 (unknown):
```unknown
Sinha_F2             24275
Sinha_E2             24227
BHF_F_Hea11192328    21714
Sinha_E8             17219
BHF_F_Hea11192324    16688
BHF_F_Hea11192326    14117
BHF_F_Hea11192325    10676
Sinha_F4             10652
BHF_F_Hea11192327     9130
BHF_F_Hea10402918     8010
Sinha_F7              6973
Sinha_D8              6714
Sinha_A1              6640
Sinha_B1              5988
BHF_F_Hea11192323     4893
Sinha_C8              4587
BHF_F_Hea10402917     3752
Sinha_G4              3389
Sinha_F8              3203
Sinha_G7              2610
Name: sangerID, dtype: int64
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/VisiumHD/no4_QC-filtering.html

**Contents:**
- Variables¶
- Read in b2c data¶
- QC and preprocessing¶
- Save¶
- Read in for plotting¶

**Examples:**

Example 1 (python):
```python
import matplotlib.pyplot as plt
import scanpy as sc
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os

import celltypist
```

Example 2 (python):
```python
adata_dir = '/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects'
sample_id = 'HEA_FOET14880396'
```

Example 3 (python):
```python
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells.h5ad')
adata
```

Example 4 (unknown):
```unknown
AnnData object with n_obs × n_vars = 134590 × 18051
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells'
    uns: 'spatial'
    obsm: 'spatial', 'spatial_cropped_150_buffer'
```

Example 5 (python):
```python
adata.obs.head()
```

Example 6 (python):
```python
adata.var["mt"] = adata.var_names.str.startswith("MT-")
sc.pp.calculate_qc_metrics(adata, qc_vars=["mt"], inplace=True)
```

Example 7 (python):
```python
fig, axs = plt.subplots(1, 4, figsize=(15, 3))
sns.histplot(adata.obs["total_counts"], kde=False, ax=axs[0])
sns.histplot(
    adata.obs["total_counts"][adata.obs["total_counts"] < 1000],
    kde=False,
    bins=40,
    ax=axs[1],
)
sns.histplot(adata.obs["n_genes_by_counts"], kde=False, bins=60, ax=axs[2])
sns.histplot(
    adata.obs["n_genes_by_counts"][adata.obs["n_genes_by_counts"] < 1000],
    kde=False,
    bins=60,
    ax=axs[3],
)
fig.suptitle('bin2cell')
fig.tight_layout()
```

Example 8 (python):
```python
fig, axs = plt.subplots(1, 2, figsize=(10, 3))
sns.histplot(
    adata.obs["total_counts"][adata.obs["total_counts"] < 8000],
    kde=False,
    bins=60,
    ax=axs[0],
)
sns.histplot(
    adata.obs["n_genes_by_counts"][adata.obs["n_genes_by_counts"] < 3500],
    kde=False,
    bins=60,
    ax=axs[1],
)
fig.suptitle('bin2cell')
fig.tight_layout()
```

Example 9 (unknown):
```unknown
AnnData object with n_obs × n_vars = 134590 × 18051
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
    uns: 'spatial'
    obsm: 'spatial', 'spatial_cropped_150_buffer'
```

Example 10 (python):
```python
plt.rcParams['figure.figsize'] = 4.5,1
# mito
sns.histplot(adata.obs['pct_counts_mt'][adata.obs['pct_counts_mt'] < 20], 
             kde=False)
plt.show()

sns.histplot(adata.obs['bin_count'], 
             kde=False)
plt.show()
sns.histplot(adata.obs['bin_count'][adata.obs['bin_count'] < 30], 
             kde=False)
plt.show()
```

Example 11 (python):
```python
# filtering out poor cuality cells/bins: n_genes
sc.pp.filter_cells(adata, min_genes=30)
print(f'shape of b2c: {adata.shape}')
```

Example 12 (unknown):
```unknown
shape of b2c: (131456, 18051)
```

Example 13 (python):
```python
# filtering out poor cuality cells/bins: bin_count (cell size)
adata = adata[adata.obs['bin_count']>3] # remove too small cells
adata = adata[adata.obs['bin_count']<60] # remove too large cells
print(f'shape of b2c: {adata.shape}')
```

Example 14 (unknown):
```unknown
shape of b2c: (129803, 18051)
```

Example 15 (python):
```python
sns.histplot(adata.obs['bin_count'], 
             kde=False)
plt.show()
```

Example 16 (python):
```python
fig, axs = plt.subplots(1, 4, figsize=(15, 3))
sns.histplot(adata.obs["total_counts"], kde=False, ax=axs[0])
sns.histplot(
    adata.obs["total_counts"][adata.obs["total_counts"] < 1000],
    kde=False,
    bins=40,
    ax=axs[1],
)
sns.histplot(adata.obs["n_genes_by_counts"], kde=False, bins=60, ax=axs[2])
sns.histplot(
    adata.obs["n_genes_by_counts"][adata.obs["n_genes_by_counts"] < 1000],
    kde=False,
    bins=60,
    ax=axs[3],
)
fig.suptitle('bin2cell')
fig.tight_layout()
```

Example 17 (python):
```python
# filtering genes
sc.pp.filter_genes(adata, min_cells=3)
print(f'shape of b2c: {adata.shape}')
```

Example 18 (unknown):
```unknown
shape of b2c: (129803, 18047)
```

Example 19 (python):
```python
# Plotting the histograms
plt.figure(figsize=(10, 3))
sns.histplot(
    adata.obs["n_genes_by_counts"][adata.obs["n_genes_by_counts"] < 4000],
    kde=False,
    bins=60,
)
plt.legend()
plt.show()
```

Example 20 (python):
```python
adata.write(f'{adata_dir}/{sample_id}_b2c_cells_filtered.h5ad')
```

Example 21 (python):
```python
f'{adata_dir}/{sample_id}_b2c_cells_filtered.h5ad'
```

Example 22 (unknown):
```unknown
'/lustre/scratch126/cellgen/team205/kk18/VisiumHD/objects/HEA_FOET14880396_b2c_cells_filtered.h5ad'
```

Example 23 (python):
```python
adata = sc.read_h5ad(f'{adata_dir}/{sample_id}_b2c_cells_filtered.h5ad')
adata
```

Example 24 (unknown):
```unknown
AnnData object with n_obs × n_vars = 129803 × 18047
    obs: 'object_id', 'bin_count', 'array_row', 'array_col', 'labels_joint_source', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_100_genes', 'pct_counts_in_top_200_genes', 'pct_counts_in_top_500_genes', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells', 'mt', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
    uns: 'spatial'
    obsm: 'spatial', 'spatial_cropped_150_buffer'
```

Example 25 (python):
```python
fig, axs = plt.subplots(1, 2, figsize=(8, 3))
sns.histplot(adata.obs["total_counts"], kde=False, ax=axs[0])
sns.histplot(adata.obs["n_genes_by_counts"], kde=False, bins=60, ax=axs[1])
fig.suptitle('bin2cell')
fig.tight_layout()
```

Example 26 (python):
```python
# median
print(f'total_counts: {np.median(adata.obs["total_counts"])}')
print(f'n_genes: {np.median(adata.obs["n_genes_by_counts"])}')
```

Example 27 (unknown):
```unknown
total_counts: 2601.3060987148215
n_genes: 1138.0
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Xenium/no1_per-sample-processing/C194-HEA-0-FFPE-1/no1_QC_filetering.html

**Contents:**
- Read in¶
- Calculate quality control metrics¶
- Filtering out low-count cells¶
- Save¶

**Examples:**

Example 1 (python):
```python
import scanpy as sc
import numpy as np
import pandas as pd
import os

import matplotlib.pyplot as plt
import seaborn as sns
```

Example 2 (unknown):
```unknown
!ls /home/kk837/rfs/rfs-teichlab-nfs-iCNyzSAaucw/jc2226/heart_team_data/xenium/20241113__133901__JK_HEA5K_Hs_heart_vessels
```

Example 3 (unknown):
```unknown
output-XETG00155__0038587__C194-HEA-0-FFPE-1__20241113__134112
output-XETG00155__0038587__Hst45-HEA-0-FFPE-1__20241113__134112
output-XETG00155__0038587__Hst60-HEA-0-FFPE-1__20241113__134112
output-XETG00155__0038592__AH13-HEA-36-FFPE-1__20241113__134112
output-XETG00155__0038592__AH14-HEA-29-FFPE-2__20241113__134112
output-XETG00155__0038592__AH14-HEA-36-FFPE-1__20241113__134112
```

Example 4 (python):
```python
sample_id = 'C194-HEA-0-FFPE-1'
out_dir = f'/home/kk837/rfs/rfs-teichlab-nfs-iCNyzSAaucw/jc2226/heart_team_data/xenium/20241113__133901__JK_HEA5K_Hs_heart_vessels/output-XETG00155__0038587__{sample_id}__20241113__134112'

adata = sc.read_10x_h5(
    filename=os.path.join(out_dir,"cell_feature_matrix.h5")
)
print(adata.X.data[:5])
adata
```

Example 5 (unknown):
```unknown
[5. 1. 1. 2. 2.]
```

Example 6 (unknown):
```unknown
AnnData object with n_obs × n_vars = 146232 × 5001
    var: 'gene_ids', 'feature_types', 'genome'
```

Example 7 (python):
```python
# add obs
df = pd.read_csv(
    os.path.join(out_dir,"cells.csv.gz")
)
df.set_index(adata.obs_names, inplace=True)
adata.obs = df.copy()
adata.obs.head()
```

Example 8 (python):
```python
adata.obsm["spatial"] = adata.obs[["x_centroid", "y_centroid"]].copy().to_numpy()
```

Example 9 (unknown):
```unknown
AnnData object with n_obs × n_vars = 146232 × 5001
    obs: 'cell_id', 'x_centroid', 'y_centroid', 'transcript_counts', 'control_probe_counts', 'genomic_control_counts', 'control_codeword_counts', 'unassigned_codeword_counts', 'deprecated_codeword_counts', 'total_counts', 'cell_area', 'nucleus_area', 'nucleus_count', 'segmentation_method'
    var: 'gene_ids', 'feature_types', 'genome'
    obsm: 'spatial'
```

Example 10 (python):
```python
sc.pl.spatial(adata,color=['total_counts'],cmap='RdPu',spot_size=20,vmax='p95')
```

Example 11 (python):
```python
/tmp/ipykernel_364135/27570213.py:1: FutureWarning: Use `squidpy.pl.spatial_scatter` instead.
  sc.pl.spatial(adata,color=['total_counts'],cmap='RdPu',spot_size=20,vmax='p95')
```

Example 12 (python):
```python
adata.obs['segmentation_method'].value_counts()
```

Example 13 (unknown):
```unknown
segmentation_method
Segmented by interior stain (18S)                       114599
Segmented by boundary stain (ATP1A1+CD45+E-Cadherin)     30718
Segmented by nucleus expansion of 5.0µm                    915
Name: count, dtype: int64
```

Example 14 (python):
```python
plt.hist(adata.obs.total_counts, bins=1000)
# plt.xlim(0,500)
plt.xscale('log')
plt.show()
```

Example 15 (python):
```python
sc.pp.calculate_qc_metrics(adata, percent_top=(10, 20, 50, 150), inplace=True)
```

Example 16 (python):
```python
# The percentage of control probes and control codewords can be calculated from adata.obs
cprobes = (
    adata.obs["control_probe_counts"].sum() / adata.obs["total_counts"].sum() * 100
)
cwords = (
    adata.obs["control_codeword_counts"].sum() / adata.obs["total_counts"].sum() * 100
)
print(f"Negative DNA probe count % : {cprobes}")
print(f"Negative decoding count % : {cwords}")
```

Example 17 (unknown):
```unknown
Negative DNA probe count % : 0.0008306279191532507
Negative decoding count % : 0.004514935328449433
```

Example 18 (python):
```python
# Next we plot the distribution of total transcripts per cell, unique transcripts per cell, area of segmented cells and the ratio of nuclei area to their cells
fig, axs = plt.subplots(1, 4, figsize=(20, 4))

axs[0].set_title("Total transcripts per cell")
sns.histplot(
    adata.obs["total_counts"],
    kde=False,
    ax=axs[0],
)

axs[1].set_title("Unique transcripts per cell")
sns.histplot(
    adata.obs["n_genes_by_counts"],
    kde=False,
    ax=axs[1],
)


axs[2].set_title("Area of segmented cells")
sns.histplot(
    adata.obs["cell_area"],
    kde=False,
    ax=axs[2],
)

axs[3].set_title("Nucleus ratio")
sns.histplot(
    adata.obs["nucleus_area"] / adata.obs["cell_area"],
    kde=False,
    ax=axs[3],
)
```

Example 19 (unknown):
```unknown
<Axes: title={'center': 'Nucleus ratio'}, ylabel='Count'>
```

Example 20 (python):
```python
plt.hist(adata.obs.n_genes_by_counts, bins=1000)
# plt.xlim(0,500)
plt.xscale('log')
plt.show()
```

Example 21 (python):
```python
sc.pp.filter_cells(adata,min_counts=50)
sc.pp.filter_cells(adata,min_genes=10)
```

Example 22 (python):
```python
adata.write(f'/home/kk837/rds/rds-teichlab-C9woKbOCf2Y/kk837/Foetal/anndata_objects/Xenium/{sample_id}_5K_filtered_raw.h5ad')
```

Example 23 (unknown):
```unknown
'C194-HEA-0-FFPE-1'
```

Example 24 (unknown):
```unknown
AnnData object with n_obs × n_vars = 144916 × 5001
    obs: 'cell_id', 'x_centroid', 'y_centroid', 'transcript_counts', 'control_probe_counts', 'genomic_control_counts', 'control_codeword_counts', 'unassigned_codeword_counts', 'deprecated_codeword_counts', 'total_counts', 'cell_area', 'nucleus_area', 'nucleus_count', 'segmentation_method', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'pct_counts_in_top_10_genes', 'pct_counts_in_top_20_genes', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_150_genes', 'n_counts', 'n_genes'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
    obsm: 'spatial'
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/Xenium/no1_per-sample-processing/Hst45-HEA-0-FFPE-1/no1_QC_filetering.html

**Contents:**
- Read in¶
- Calculate quality control metrics¶
- Filtering out low-count cells¶
- Save¶

**Examples:**

Example 1 (python):
```python
import scanpy as sc
import numpy as np
import pandas as pd
import os

import matplotlib.pyplot as plt
import seaborn as sns
```

Example 2 (unknown):
```unknown
!ls /home/kk837/rfs/rfs-teichlab-nfs-iCNyzSAaucw/jc2226/heart_team_data/xenium/20241113__133901__JK_HEA5K_Hs_heart_vessels
```

Example 3 (unknown):
```unknown
output-XETG00155__0038587__C194-HEA-0-FFPE-1__20241113__134112
output-XETG00155__0038587__Hst45-HEA-0-FFPE-1__20241113__134112
output-XETG00155__0038587__Hst60-HEA-0-FFPE-1__20241113__134112
output-XETG00155__0038592__AH13-HEA-36-FFPE-1__20241113__134112
output-XETG00155__0038592__AH14-HEA-29-FFPE-2__20241113__134112
output-XETG00155__0038592__AH14-HEA-36-FFPE-1__20241113__134112
```

Example 4 (python):
```python
sample_id = 'Hst45-HEA-0-FFPE-1'
out_dir = f'/home/kk837/rfs/rfs-teichlab-nfs-iCNyzSAaucw/jc2226/heart_team_data/xenium/20241113__133901__JK_HEA5K_Hs_heart_vessels/output-XETG00155__0038587__{sample_id}__20241113__134112'

adata = sc.read_10x_h5(
    filename=os.path.join(out_dir,"cell_feature_matrix.h5")
)
print(adata.X.data[:5])
adata
```

Example 5 (unknown):
```unknown
[1. 1. 1. 1. 4.]
```

Example 6 (unknown):
```unknown
AnnData object with n_obs × n_vars = 219198 × 5001
    var: 'gene_ids', 'feature_types', 'genome'
```

Example 7 (python):
```python
# add obs
df = pd.read_csv(
    os.path.join(out_dir,"cells.csv.gz")
)
df.set_index(adata.obs_names, inplace=True)
adata.obs = df.copy()
adata.obs.head()
```

Example 8 (python):
```python
adata.obsm["spatial"] = adata.obs[["x_centroid", "y_centroid"]].copy().to_numpy()
```

Example 9 (unknown):
```unknown
AnnData object with n_obs × n_vars = 219198 × 5001
    obs: 'cell_id', 'x_centroid', 'y_centroid', 'transcript_counts', 'control_probe_counts', 'genomic_control_counts', 'control_codeword_counts', 'unassigned_codeword_counts', 'deprecated_codeword_counts', 'total_counts', 'cell_area', 'nucleus_area', 'nucleus_count', 'segmentation_method'
    var: 'gene_ids', 'feature_types', 'genome'
    obsm: 'spatial'
```

Example 10 (python):
```python
sc.pl.spatial(adata,color=['total_counts'],cmap='RdPu',spot_size=20,vmax='p95')
```

Example 11 (python):
```python
/tmp/ipykernel_184408/27570213.py:1: FutureWarning: Use `squidpy.pl.spatial_scatter` instead.
  sc.pl.spatial(adata,color=['total_counts'],cmap='RdPu',spot_size=20,vmax='p95')
```

Example 12 (python):
```python
adata.obs['segmentation_method'].value_counts()
```

Example 13 (unknown):
```unknown
segmentation_method
Segmented by interior stain (18S)                       213482
Segmented by boundary stain (ATP1A1+CD45+E-Cadherin)      2861
Segmented by nucleus expansion of 5.0µm                   2855
Name: count, dtype: int64
```

Example 14 (python):
```python
plt.hist(adata.obs.total_counts, bins=1000)
# plt.xlim(0,500)
plt.xscale('log')
plt.show()
```

Example 15 (python):
```python
sc.pp.calculate_qc_metrics(adata, percent_top=(10, 20, 50, 150), inplace=True)
```

Example 16 (python):
```python
# The percentage of control probes and control codewords can be calculated from adata.obs
cprobes = (
    adata.obs["control_probe_counts"].sum() / adata.obs["total_counts"].sum() * 100
)
cwords = (
    adata.obs["control_codeword_counts"].sum() / adata.obs["total_counts"].sum() * 100
)
print(f"Negative DNA probe count % : {cprobes}")
print(f"Negative decoding count % : {cwords}")
```

Example 17 (unknown):
```unknown
Negative DNA probe count % : 0.0009415129050999326
Negative decoding count % : 0.004741297553449501
```

Example 18 (python):
```python
# Next we plot the distribution of total transcripts per cell, unique transcripts per cell, area of segmented cells and the ratio of nuclei area to their cells
fig, axs = plt.subplots(1, 4, figsize=(20, 4))

axs[0].set_title("Total transcripts per cell")
sns.histplot(
    adata.obs["total_counts"],
    kde=False,
    ax=axs[0],
)

axs[1].set_title("Unique transcripts per cell")
sns.histplot(
    adata.obs["n_genes_by_counts"],
    kde=False,
    ax=axs[1],
)


axs[2].set_title("Area of segmented cells")
sns.histplot(
    adata.obs["cell_area"],
    kde=False,
    ax=axs[2],
)

axs[3].set_title("Nucleus ratio")
sns.histplot(
    adata.obs["nucleus_area"] / adata.obs["cell_area"],
    kde=False,
    ax=axs[3],
)
```

Example 19 (unknown):
```unknown
<Axes: title={'center': 'Nucleus ratio'}, ylabel='Count'>
```

Example 20 (python):
```python
plt.hist(adata.obs.n_genes_by_counts, bins=1000)
# plt.xlim(0,500)
plt.xscale('log')
plt.show()
```

Example 21 (python):
```python
sc.pp.filter_cells(adata,min_counts=50)
sc.pp.filter_cells(adata,min_genes=10)
```

Example 22 (python):
```python
adata.write(f'/home/kk837/rds/rds-teichlab-C9woKbOCf2Y/kk837/Foetal/anndata_objects/Xenium/{sample_id}_5K_filtered_raw.h5ad')
```

Example 23 (unknown):
```unknown
'Hst45-HEA-0-FFPE-1'
```

Example 24 (unknown):
```unknown
AnnData object with n_obs × n_vars = 217361 × 5001
    obs: 'cell_id', 'x_centroid', 'y_centroid', 'transcript_counts', 'control_probe_counts', 'genomic_control_counts', 'control_codeword_counts', 'unassigned_codeword_counts', 'deprecated_codeword_counts', 'total_counts', 'cell_area', 'nucleus_area', 'nucleus_count', 'segmentation_method', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'log1p_total_counts', 'pct_counts_in_top_10_genes', 'pct_counts_in_top_20_genes', 'pct_counts_in_top_50_genes', 'pct_counts_in_top_150_genes', 'n_counts', 'n_genes'
    var: 'gene_ids', 'feature_types', 'genome', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
    obsm: 'spatial'
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/QC/Foetal_scRNA_No2_QC-filter.html

**Contents:**
- Import modules¶
- Functions¶
- Read in anndata¶
- Remove doublets¶
- Remove low quality cells¶
- Compute QC stats¶
- Remove cells exceeding the ribo/mito threshold¶
- Calculate RBC score¶
- Remove RBCs¶
- Save¶

**Examples:**

Example 1 (python):
```python
import numpy as np
import pandas as pd
import scanpy as sc
import matplotlib.pyplot as plt
import scrublet as scr
import session_info
```

Example 2 (python):
```python
sc.settings.verbosity = 3
sc.settings.set_figure_params(dpi = 120, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.3.4
numpy               1.20.1
pandas              1.1.4
scanpy              1.7.1
scrublet            NA
session_info        1.0.0
-----
```

Example 4 (unknown):
```unknown
PIL                 8.1.2
anndata             0.7.5
anyio               NA
attr                20.3.0
babel               2.9.0
backcall            0.2.0
brotli              NA
cairo               1.20.0
certifi             2020.12.05
cffi                1.14.5
chardet             4.0.0
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.0
dask                2021.03.1
dateutil            2.8.1
decorator           4.4.2
fsspec              0.8.7
get_version         2.1
google              NA
h5py                2.10.0
idna                2.10
igraph              0.8.3
ipykernel           5.5.0
ipython_genutils    0.2.0
jedi                0.18.0
jinja2              2.11.3
joblib              1.0.1
json5               NA
jsonschema          3.2.0
jupyter_server      1.4.1
jupyterlab_server   2.3.0
kiwisolver          1.3.1
legacy_api_wrap     0.0.0
leidenalg           0.8.3
llvmlite            0.34.0
louvain             0.7.0
markupsafe          1.1.1
mpl_toolkits        NA
natsort             7.1.1
nbclassic           NA
nbformat            5.1.2
numba               0.51.2
numexpr             2.7.3
packaging           20.9
parso               0.8.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
prometheus_client   NA
prompt_toolkit      3.0.16
psutil              5.8.0
ptyprocess          0.7.0
pvectorc            NA
pyarrow             0.16.0
pygments            2.8.0
pyparsing           2.4.7
pyrsistent          NA
pytoml              NA
pytz                2021.1
requests            2.25.1
ruamel              NA
scipy               1.6.1
send2trash          NA
setuptools_scm      NA
simplejson          3.17.2
sinfo               0.3.1
six                 1.15.0
sklearn             0.22
sniffio             1.2.0
socks               1.7.1
sphinxcontrib       NA
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
tlz                 0.11.0
toolz               0.11.1
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
uritemplate         3.0.1
urllib3             1.26.3
wcwidth             0.2.5
yaml                5.1.2
zmq                 22.0.3
```

Example 5 (unknown):
```unknown
-----
IPython             7.21.0
jupyter_client      6.1.11
jupyter_core        4.7.1
jupyterlab          3.0.9
notebook            6.2.0
-----
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0]
Linux-4.15.0-158-generic-x86_64-with-glibc2.10
-----
Session information updated at 2022-12-30 10:13
```

Example 6 (python):
```python
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/QC/scRNA_foetal_prefilter.h5ad')
adata
```

Example 7 (unknown):
```unknown
AnnData object with n_obs × n_vars = 205457 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-10', 'ambient_expression-11', 'ambient_expression-12', 'ambient_expression-13', 'ambient_expression-14', 'ambient_expression-15', 'ambient_expression-16', 'ambient_expression-17', 'ambient_expression-18', 'ambient_expression-19', 'ambient_expression-2', 'ambient_expression-3', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7', 'ambient_expression-8', 'ambient_expression-9'
    obsm: 'latent_gene_encoding'
```

Example 8 (python):
```python
adata.obs['sangerID'].value_counts()
```

Example 9 (unknown):
```unknown
Sinha_F2             24275
Sinha_E2             24227
BHF_F_Hea11192328    21714
Sinha_E8             17219
BHF_F_Hea11192324    16688
BHF_F_Hea11192326    14117
BHF_F_Hea11192325    10676
Sinha_F4             10652
BHF_F_Hea11192327     9130
BHF_F_Hea10402918     8010
Sinha_F7              6973
Sinha_D8              6714
Sinha_A1              6640
Sinha_B1              5988
BHF_F_Hea11192323     4893
Sinha_C8              4587
BHF_F_Hea10402917     3752
Sinha_G4              3389
Sinha_F8              3203
Sinha_G7              2610
Name: sangerID, dtype: int64
```

Example 10 (python):
```python
adata.obs['scrublet_score'].hist(bins=100)
```

Example 11 (unknown):
```unknown
<AxesSubplot:>
```

Example 12 (python):
```python
# set threshold (arbitrary)
dbl_score_thresh=0.15

prop = np.sum(adata.obs['scrublet_score']>=dbl_score_thresh)/len(adata)
print(f'proportion of doublets: {prop}')
```

Example 13 (unknown):
```unknown
proportion of doublets: 0.15717644081243276
```

Example 14 (python):
```python
# Save doublet ids
doublets = adata.obs_names[(adata.obs['scrublet_score']<dbl_score_thresh)==False]
with open('/nfs/team205/heart/anndata_objects/Foetal/QC/scRNA_rm-doublets.txt', 'w') as f:
    for line in doublets:
        f.write(line)
        f.write('\n')

# Remove doublets
adata = adata[adata.obs['scrublet_score']<dbl_score_thresh]
print(adata.shape)
```

Example 15 (unknown):
```unknown
(173164, 36601)
```

Example 16 (python):
```python
sc.pp.filter_cells(adata, min_genes=500, inplace=True)
sc.pp.filter_cells(adata, min_counts=1000, inplace=True)
print(adata.shape)
```

Example 17 (unknown):
```unknown
filtered out 56470 cells that have less than 500 genes expressed
Trying to set attribute `.obs` of view, copying.
filtered out 1964 cells that have less than 1000 counts
```

Example 18 (unknown):
```unknown
(114730, 36601)
```

Example 19 (python):
```python
adata.obs['sangerID'].value_counts()
```

Example 20 (unknown):
```unknown
BHF_F_Hea11192328    12134
BHF_F_Hea11192324    10697
BHF_F_Hea11192326     9926
Sinha_E2              8388
Sinha_F4              8149
Sinha_E8              8089
BHF_F_Hea11192325     7532
Sinha_F2              6849
BHF_F_Hea11192327     5895
Sinha_D8              5213
Sinha_F7              4983
BHF_F_Hea10402918     4402
Sinha_A1              4202
Sinha_C8              3745
BHF_F_Hea11192323     3533
Sinha_B1              3361
Sinha_G4              2404
BHF_F_Hea10402917     2238
Sinha_G7              1660
Sinha_F8              1330
Name: sangerID, dtype: int64
```

Example 21 (python):
```python
adata.var['mt'] = adata.var['gene_name'].str.startswith('MT-')  # annotate the group of mitochondrial genes as 'mt'
adata.var['ribo'] = adata.var['gene_name'].str.startswith('RPS','RPL')  # annotate the group of ribosomal genes as 'mt'
sc.pp.calculate_qc_metrics(adata, qc_vars=['mt','ribo'], percent_top=None, log1p=True, inplace=True)
```

Example 22 (python):
```python
plt.rcParams['figure.figsize'] = [3, 2]

print('### Sanger ###')
fig,axs=plt.subplots(nrows=1,ncols=3,figsize=(10,3))
sc.pl.scatter(adata[adata.obs['facility']=='Sanger'], x='total_counts', y='n_genes', show=False, ax=axs[0])
sc.pl.scatter(adata[adata.obs['facility']=='Sanger'], x='total_counts', y='pct_counts_mt', show=False, ax=axs[1])
sc.pl.scatter(adata[adata.obs['facility']=='Sanger'], x='total_counts', y='pct_counts_ribo', show=False, ax=axs[2])

fig.tight_layout()
plt.show()

plt.rcParams['figure.figsize'] = [3, 2]

print('### CSCI ###')
fig,axs=plt.subplots(nrows=1,ncols=3,figsize=(10,3))
sc.pl.scatter(adata[adata.obs['facility']=='CSCI'], x='total_counts', y='n_genes', show=False, ax=axs[0])
sc.pl.scatter(adata[adata.obs['facility']=='CSCI'], x='total_counts', y='pct_counts_mt', show=False, ax=axs[1])
sc.pl.scatter(adata[adata.obs['facility']=='CSCI'], x='total_counts', y='pct_counts_ribo', show=False, ax=axs[2])

fig.tight_layout()
plt.show()
```

Example 23 (unknown):
```unknown
### Sanger ###
```

Example 24 (unknown):
```unknown
### CSCI ###
```

Example 25 (python):
```python
plt.rcParams['figure.figsize'] = [2, 2]
adata = adata[adata.obs['pct_counts_mt'] < 20, :]
adata = adata[adata.obs['pct_counts_ribo'] < 20, :]
sc.pl.violin(adata, ['n_genes', 'total_counts','log1p_total_counts', 'pct_counts_mt', 'pct_counts_ribo'],jitter = 0.2, multi_panel = True)
```

Example 26 (python):
```python
sc.settings.set_figure_params(dpi = 120, color_map = 'RdPu')
plt.rcParams['figure.figsize'] = [8, 2]
sc.pl.violin(adata,keys=['log1p_total_counts'],groupby='sangerID',rotation=90,jitter=False)
```

Example 27 (unknown):
```unknown
Trying to set attribute `.uns` of view, copying.
```

Example 28 (python):
```python
print(adata.shape)
adata.obs['sangerID'].value_counts()
```

Example 29 (unknown):
```unknown
(113279, 36601)
```

Example 30 (unknown):
```unknown
BHF_F_Hea11192328    12086
BHF_F_Hea11192324    10568
BHF_F_Hea11192326     9896
Sinha_E2              8327
Sinha_E8              8079
Sinha_F4              8052
BHF_F_Hea11192325     7515
Sinha_F2              6807
BHF_F_Hea11192327     5869
Sinha_D8              4987
Sinha_F7              4837
BHF_F_Hea10402918     4371
Sinha_A1              4118
Sinha_C8              3551
BHF_F_Hea11192323     3420
Sinha_B1              3293
Sinha_G4              2352
BHF_F_Hea10402917     2232
Sinha_G7              1623
Sinha_F8              1296
Name: sangerID, dtype: int64
```

Example 31 (python):
```python
adata_c = adata.copy()

adata_c.var.set_index('gene_name',inplace=True)
adata_c.var_names = adata_c.var_names.astype('str')
adata_c.var_names_make_unique()

sc.pp.filter_genes(adata_c, min_cells=3)
sc.pp.normalize_total(adata_c, target_sum=1e4)
sc.pp.log1p(adata_c)
```

Example 32 (unknown):
```unknown
filtered out 5479 genes that are detected in less than 3 cells
normalizing counts per cell
    finished (0:00:01)
```

Example 33 (python):
```python
# scoring hemoglobin genes
# HGNC, Hemoglobin subunits, groupID:940
genes=['HBA1','HBA2','HBB','HBD','HBE1','HBG1','HBG2','HBM','HBQ1','HBZ']
sc.tl.score_genes(adata_c,gene_list=genes,score_name='HB_score')
```

Example 34 (python):
```python
computing score 'HB_score'
    finished: added
    'HB_score', score of gene set (adata.obs).
    199 total control genes are used. (0:00:04)
```

Example 35 (python):
```python
len(adata_c.obs['sangerID'].unique())
```

Example 36 (python):
```python
sc.settings.set_figure_params(dpi = 80, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')

fig,axs=plt.subplots(nrows=4,ncols=5,figsize=(15,7))

for i,ID in enumerate(adata_c.obs['sangerID'].unique()):
    sample = adata_c[adata_c.obs['sangerID']==ID]
    axs[i//5,i%5].hist(sample.obs['HB_score'],bins=100)
    axs[i//5,i%5].set_title(ID)

fig.tight_layout()
plt.show()
```

Example 37 (python):
```python
# add HB score to original adata
adata.obs.loc[adata_c.obs_names,'HB_score']=adata_c.obs['HB_score']
```

Example 38 (python):
```python
HBscore_thresh=2

adata=adata[adata.obs['HB_score']<HBscore_thresh]
print(adata.shape)
```

Example 39 (unknown):
```unknown
(100463, 36601)
```

Example 40 (python):
```python
# save
adata.write('/nfs/team205/heart/anndata_objects/Foetal/QC/scRNA_foetal.h5ad')
adata
```

Example 41 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 100463 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval', 'n_genes', 'n_counts', 'n_genes_by_counts', 'log1p_n_genes_by_counts', 'total_counts', 'log1p_total_counts', 'total_counts_mt', 'log1p_total_counts_mt', 'pct_counts_mt', 'total_counts_ribo', 'log1p_total_counts_ribo', 'pct_counts_ribo', 'HB_score'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-10', 'ambient_expression-11', 'ambient_expression-12', 'ambient_expression-13', 'ambient_expression-14', 'ambient_expression-15', 'ambient_expression-16', 'ambient_expression-17', 'ambient_expression-18', 'ambient_expression-19', 'ambient_expression-2', 'ambient_expression-3', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7', 'ambient_expression-8', 'ambient_expression-9', 'mt', 'ribo', 'n_cells_by_counts', 'mean_counts', 'log1p_mean_counts', 'pct_dropout_by_counts', 'total_counts', 'log1p_total_counts'
    uns: 'sangerID_colors'
    obsm: 'latent_gene_encoding'
```

Example 42 (python):
```python
adata.obs['sangerID'].value_counts()
```

Example 43 (unknown):
```unknown
BHF_F_Hea11192328    12056
BHF_F_Hea11192324    10553
BHF_F_Hea11192326     9876
Sinha_F4              7621
BHF_F_Hea11192325     7512
Sinha_E8              7511
BHF_F_Hea11192327     5865
Sinha_F7              4545
Sinha_F2              4495
Sinha_D8              4377
BHF_F_Hea10402918     4340
Sinha_E2              3913
BHF_F_Hea11192323     3418
Sinha_C8              2989
Sinha_A1              2604
BHF_F_Hea10402917     2227
Sinha_B1              2145
Sinha_G4              2030
Sinha_G7              1372
Sinha_F8              1014
Name: sangerID, dtype: int64
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/QC/Demultiplex/No3_SOC-CMO-comparison-with-GEX.html

**Contents:**
- Read in ID metadata¶
- Read in Multiome data¶
- Add SOC result¶
- Add CMO result¶
- Comparison of SOC and CMO result with confusion matrix¶
- Comparison of SOC and CMO result with GEX (UMAP)¶
- "CMO | ratio" of matched and unmatched nuclei¶
- setting a threshold with matched and unmatched nuclei¶
  - Option 1: to be max(tpr-fpr)¶
  - Option 2: fpr lower than 0.05¶
- Additional work¶
  - Only SOC-assigned ones¶

--> similar patter between SOC and CMO

--> CMO is lacking assignments in the left island

--> CMO is over-assigning nuclei in the lower right part of the largest island

--> Unmatched samples has smaller "prop_delta" value, which suggest this metrics is usable for rescuing nuclei using CMO result (for those nuclei whose donor were not assigned by SOC)

27498 rows × 35 columns

194705 rows × 35 columns

73111 rows × 37 columns

**Examples:**

Example 1 (python):
```python
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import numpy as np
import pandas as pd
import scanpy as sc
import session_info
import matplotlib.pyplot as plt
import seaborn as sns


import os
```

Example 2 (unknown):
```unknown
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.5.2
numpy               1.21.5
pandas              1.3.5
scanpy              1.8.2
seaborn             0.11.1
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                 9.1.1
anndata             0.8.0
asciitree           NA
backcall            0.2.0
beta_ufunc          NA
binom_ufunc         NA
cached_property     1.5.1
cairo               1.20.0
cffi                1.14.4
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.2
dask                2022.01.0
dateutil            2.8.1
decorator           4.4.2
defusedxml          0.6.0
fasteners           0.17.3
fsspec              2022.01.0
google              NA
h5py                3.1.0
igraph              0.10.2
importlib_metadata  1.7.0
ipykernel           5.4.2
ipython_genutils    0.2.0
ipywidgets          7.6.3
jedi                0.17.2
jinja2              3.1.2
joblib              1.0.0
kiwisolver          1.3.1
leidenalg           0.9.0
llvmlite            0.35.0
louvain             0.7.0
markupsafe          2.1.1
mpl_toolkits        NA
msgpack             1.0.3
natsort             7.1.0
nbinom_ufunc        NA
numba               0.52.0
numcodecs           0.9.1
numexpr             2.7.2
packaging           21.3
parso               0.7.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
plotly              5.10.0
prompt_toolkit      3.0.10
psutil              5.9.0
ptyprocess          0.7.0
pyarrow             9.0.0
pygments            2.13.0
pyparsing           2.4.7
pytz                2020.5
scipy               1.7.3
setuptools_scm      NA
sinfo               0.3.1
six                 1.14.0
sklearn             1.0.2
sphinxcontrib       NA
statsmodels         0.12.1
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
threadpoolctl       2.1.0
tlz                 0.11.2
toolz               0.11.2
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
wcwidth             0.2.5
yaml                5.4.1
zarr                2.11.3
zipp                NA
zmq                 21.0.1
zope                NA
```

Example 5 (unknown):
```unknown
-----
IPython             7.19.0
jupyter_client      6.1.11
jupyter_core        4.7.0
notebook            6.2.0
-----
Python 3.7.9 | packaged by conda-forge | (default, Dec  9 2020, 21:08:20) [GCC 9.3.0]
Linux-4.15.0-112-generic-x86_64-with-debian-bullseye-sid
-----
Session information updated at 2023-02-08 18:37
```

Example 6 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 7 (python):
```python
multiplexed_samples = pd.read_csv('/nfs/team205/heart/mapped/cellranger-arc200/csv/multiome-foetal_20221221.csv')
multiplexed_samples

# read in metadata
meta = pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20221223.csv', sep = ',', index_col = None)
meta = meta[meta['Publication']=='Foetal']
meta = meta[['sangerID', 'combinedID',
       'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender',
       'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed',
       'status']]

# select samples
meta = meta[meta['combinedID'].isin(multiplexed_samples['combinedID'])]

# select Multiome-ATAC
meta = meta[meta['modality']=='Multiome-ATAC']

# reindex
meta.index = range(len(meta))

pd.set_option('max_colwidth', None)
meta[['combinedID',
       'donor', 'region', 'age', 'gender']]
```

Example 8 (unknown):
```unknown
multiplexed_samples = ['7089STDY13216920_BHF_F_Hea13242527','7089STDY13216921_BHF_F_Hea13242528',
                       '7089STDY13216922_BHF_F_Hea13242529','7089STDY13216923_BHF_F_Hea13242530']
```

Example 9 (python):
```python
# post-cellbender
adata = sc.read_h5ad('/nfs/team205/heart/anndata_objects/Foetal/QC/multiome_RNA_foetal_prefilter.h5ad')

# only multiplexed data (plus relevant single genotype samples)
adata = adata[adata.obs['combinedID'].isin(list(meta['combinedID']))]

adata
```

Example 10 (unknown):
```unknown
View of AnnData object with n_obs × n_vars = 194705 × 36601
    obs: 'latent_RT_efficiency', 'latent_cell_probability', 'latent_scale', 'sangerID', 'combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age', 'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x', 'flushed', 'scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval'
    var: 'gene_name', 'feature_type', 'ambient_expression-0', 'ambient_expression-1', 'ambient_expression-10', 'ambient_expression-11', 'ambient_expression-12', 'ambient_expression-13', 'ambient_expression-14', 'ambient_expression-15', 'ambient_expression-16', 'ambient_expression-17', 'ambient_expression-18', 'ambient_expression-19', 'ambient_expression-2', 'ambient_expression-20', 'ambient_expression-21', 'ambient_expression-22', 'ambient_expression-23', 'ambient_expression-24', 'ambient_expression-25', 'ambient_expression-26', 'ambient_expression-27', 'ambient_expression-28', 'ambient_expression-29', 'ambient_expression-3', 'ambient_expression-30', 'ambient_expression-31', 'ambient_expression-32', 'ambient_expression-33', 'ambient_expression-34', 'ambient_expression-35', 'ambient_expression-36', 'ambient_expression-37', 'ambient_expression-4', 'ambient_expression-5', 'ambient_expression-6', 'ambient_expression-7', 'ambient_expression-8', 'ambient_expression-9'
    obsm: 'latent_gene_encoding'
```

Example 11 (python):
```python
adata.obs['combinedID'].value_counts()
```

Example 12 (unknown):
```unknown
7089STDY13216924_BHF_F_Hea13242531    40000
7089STDY13216925_BHF_F_Hea13242532    30453
7089STDY13216920_BHF_F_Hea13242527    27498
7089STDY13216927_BHF_F_Hea13242534    24666
7089STDY13216926_BHF_F_Hea13242533    24602
7089STDY13216921_BHF_F_Hea13242528    24211
7089STDY13216922_BHF_F_Hea13242529    21402
7089STDY13216923_BHF_F_Hea13242530     1873
Name: combinedID, dtype: int64
```

Example 13 (python):
```python
adata.obs[adata.obs['combinedID'].isin(multiplexed_samples)]['combinedID'].value_counts()
```

Example 14 (unknown):
```unknown
7089STDY13216920_BHF_F_Hea13242527    27498
7089STDY13216921_BHF_F_Hea13242528    24211
7089STDY13216922_BHF_F_Hea13242529    21402
7089STDY13216923_BHF_F_Hea13242530     1873
7089STDY13216924_BHF_F_Hea13242531        0
7089STDY13216925_BHF_F_Hea13242532        0
7089STDY13216926_BHF_F_Hea13242533        0
7089STDY13216927_BHF_F_Hea13242534        0
Name: combinedID, dtype: int64
```

Example 15 (python):
```python
# read in
soc = pd.read_csv('/nfs/team205/heart/mapped/cellranger-arc200/SOC_out/SOC-clusters_multiome-foetal_20221221.csv',
                 index_col=0)
print(soc.shape)
soc.head()
```

Example 16 (unknown):
```unknown
(74984, 10)
```

Example 17 (python):
```python
# add
col_to_add = ['status', 'assignment', 'log_prob_singleton',
               'log_prob_doublet', 'cluster0', 'cluster1', 'cluster2',
               'donor_assignment']
adata.obs[[f'SOC | {x}' for x in col_to_add]] = soc[col_to_add].reindex(adata.obs_names)
adata.obs.head()
```

Example 18 (unknown):
```unknown
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/ipykernel_launcher.py:5: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
  """
```

Example 19 (python):
```python
adata.obs['SOC | donor_assignment'].value_counts()
```

Example 20 (unknown):
```unknown
Hst39               36323
Hst40               20246
donor_unassigned    13322
Hst36                5093
Name: SOC | donor_assignment, dtype: int64
```

Example 21 (python):
```python
adata.obs[adata.obs['SOC | donor_assignment']!='donor_unassigned']['SOC | donor_assignment'].value_counts()
```

Example 22 (unknown):
```unknown
Hst39    36323
Hst40    20246
Hst36     5093
Name: SOC | donor_assignment, dtype: int64
```

Example 23 (python):
```python
adata.obs['SOC | status'].value_counts()
```

Example 24 (javascript):
```javascript
singlet       63170
doublet        8383
unassigned     3431
Name: SOC | status, dtype: int64
```

Example 25 (python):
```python
# read in
cmo = pd.read_csv('/nfs/team205/heart/mapped/CMO/result_summary/CMO-result_multiome-foetal_20230207.csv',
                 index_col=0)
print(cmo.shape)
cmo.head()
```

Example 26 (python):
```python
# add
col_to_add = ['ratio','top_count', 'donor_assignment', 'n_donor_assigned','Hst40', 'Hst39', 'Hst36']
adata.obs[[f'CMO | {x}' for x in col_to_add]] = cmo[col_to_add].reindex(adata.obs_names)
adata.obs.head()
```

Example 27 (python):
```python
# nuclei number
df = adata.obs[adata.obs['combinedID'].isin(multiplexed_samples)]
pd.crosstab(df['SOC | donor_assignment'],df['CMO | donor_assignment'])
```

Example 28 (python):
```python
# nuclei number in each sample
df_cmo = adata.obs[adata.obs['combinedID'].isin(multiplexed_samples)]
for sam in multiplexed_samples:
    print(f'##### {sam} #####')
    df = df_cmo[df_cmo['combinedID']==sam]
    print(pd.crosstab(df['SOC | donor_assignment'],df['CMO | donor_assignment']))
    print('')
```

Example 29 (unknown):
```unknown
##### 7089STDY13216920_BHF_F_Hea13242527 #####
CMO | donor_assignment  Hst39  Hst40  Hst40,Hst39
SOC | donor_assignment                           
Hst39                    7777   8886            7
Hst40                     177   6716            1
donor_unassigned          603   3331            0

##### 7089STDY13216921_BHF_F_Hea13242528 #####
CMO | donor_assignment  Hst39  Hst40  Hst40,Hst39
SOC | donor_assignment                           
Hst39                    7675   7182           10
Hst40                     129   5815            0
donor_unassigned          531   2869            0

##### 7089STDY13216922_BHF_F_Hea13242529 #####
CMO | donor_assignment  Hst36  Hst39  Hst40  Hst40,Hst36  Hst40,Hst39
SOC | donor_assignment                                               
Hst36                    3559     73   1459            2            0
Hst39                     444   1692   2649            0            1
Hst40                     362     72   6974            0            0
donor_unassigned         1101    341   2673            0            0

##### 7089STDY13216923_BHF_F_Hea13242530 #####
CMO | donor_assignment  Hst36  Hst39  Hst40
SOC | donor_assignment                     
donor_unassigned          394     44   1435
```

Example 30 (python):
```python
%%time
adata_test = adata.copy()

# remove doublets
dbl_score_thresh=0.15
adata_test = adata_test[adata_test.obs['scrublet_score']<dbl_score_thresh]
# select regions
adata_test = adata_test[adata_test.obs['region'].isin([
    'Heart',
    'Heart (Hst36), Heart-wo-nodes (Hst39), Heart-wo-nodes (Hst40)',
    'Heart-wo-nodes'])]

# sc.pp.filter_cells(adata_test, min_genes=200)
sc.pp.filter_genes(adata_test, min_cells=3)

# QC metrics
adata_test.var['mt'] = adata_test.var_names.str.startswith('MT-')  # annotate the group of mitochondrial genes as 'mt'
sc.pp.calculate_qc_metrics(adata_test, qc_vars=['mt'], percent_top=None, log1p=True, inplace=True)
## adata_test = adata_test[adata_test.obs.pct_counts_mt < 5, :]

# log-normalise
sc.pp.normalize_total(adata_test, target_sum=1e4)
sc.pp.log1p(adata_test)

# HVGs
sc.pp.highly_variable_genes(adata_test, min_mean=0.0125, max_mean=3, min_disp=0.5)
print(f'n_hvgs: {sum(adata_test.var.highly_variable)}')
adata_test.raw = adata_test
adata_test = adata_test[:, adata_test.var.highly_variable]

# scale, pce, neighbors, umap
## sc.pp.regress_out(adata_test, ['total_counts', 'pct_counts_mt'])
sc.pp.scale(adata_test, max_value=10)
sc.tl.pca(adata_test, svd_solver='arpack')
sc.pp.neighbors(adata_test)
sc.tl.umap(adata_test, min_dist=0.1)
```

Example 31 (python):
```python
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/scanpy/preprocessing/_simple.py:251: ImplicitModificationWarning: Trying to modify attribute `.var` of view, initializing view as actual.
  adata.var['n_cells'] = number
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/scanpy/preprocessing/_normalization.py:182: UserWarning: Some cells have zero counts
  warn(UserWarning('Some cells have zero counts'))
```

Example 32 (unknown):
```unknown
n_hvgs: 5536
```

Example 33 (python):
```python
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/scanpy/preprocessing/_simple.py:843: UserWarning: Revieved a view of an AnnData. Making a copy.
  view_to_actual(adata)
```

Example 34 (unknown):
```unknown
CPU times: user 8min 4s, sys: 3min 11s, total: 11min 16s
Wall time: 2min 59s
```

Example 35 (python):
```python
fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(16,4)) 
sc.pl.umap(adata_test, color = ['combinedID',], show=False, ax=axs[0], size=0.3)
sc.pl.umap(adata_test, color = ['combinedID',], groups=['7089STDY13216922_BHF_F_Hea13242529','7089STDY13216923_BHF_F_Hea13242530'], show=False, ax=axs[1])
plt.tight_layout()
```

Example 36 (python):
```python
adata_test.obs['multiplexed'] = 'No'
multi_obsnames = adata_test.obs_names[adata_test.obs['combinedID'].isin(multiplexed_samples)]
adata_test.obs.loc[multi_obsnames,'multiplexed'] = 'Yes'

fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(9,4)) 
sc.pl.umap(adata_test, color = ['multiplexed'], groups='No', wspace=0.5, size=2, show=False, ax=axs[0])
sc.pl.umap(adata_test, color = ['multiplexed'], groups='Yes', wspace=0.5, size=2, show=False, ax=axs[1])
plt.tight_layout()
```

Example 37 (python):
```python
adata_test.uns['donor_colors'] = ['#1f77b4', '#d62728', '#ff7f0e', '#279e68']
sc.pl.umap(adata_test, color = ['donor','SOC | donor_assignment','CMO | donor_assignment',], wspace=0.6)
```

Example 38 (python):
```python
don = 'Hst36'
size = 3

fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(14,4)) 
sc.pl.umap(adata_test, color = ['donor'], groups=don, size=size, show=False, ax=axs[0], title='donor, singleplex sample')
sc.pl.umap(adata_test, color = ['SOC | donor_assignment'], groups=don, size=size, show=False, ax=axs[1])
sc.pl.umap(adata_test, color = ['CMO | donor_assignment'], groups=don, size=size, show=False, ax=axs[2])
plt.tight_layout()
```

Example 39 (python):
```python
don = 'Hst39'
size = 3

fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(14,4)) 
sc.pl.umap(adata_test, color = ['donor'], groups=don, size=size, show=False, ax=axs[0], title='donor, singleplex sample')
sc.pl.umap(adata_test, color = ['SOC | donor_assignment'], groups=don, size=size, show=False, ax=axs[1])
sc.pl.umap(adata_test, color = ['CMO | donor_assignment'], groups=don, size=size, show=False, ax=axs[2])
plt.tight_layout()
```

Example 40 (python):
```python
don = 'Hst40'
size = 3

fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(14,4)) 
sc.pl.umap(adata_test, color = ['donor'], groups=don, size=size, show=False, ax=axs[0], title='donor, singleplex sample')
sc.pl.umap(adata_test, color = ['SOC | donor_assignment'], groups=don, size=size, show=False, ax=axs[1])
sc.pl.umap(adata_test, color = ['CMO | donor_assignment'], groups=don, size=size, show=False, ax=axs[2])
plt.tight_layout()
```

Example 41 (python):
```python
sc.pl.umap(adata_test, color = ['CMO | prop_delta',], wspace=0.5)
```

Example 42 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]

pd.crosstab(df['SOC | donor_assignment'],df['CMO | donor_assignment'])
```

Example 43 (python):
```python
# new labels for the comparison
matched_obsnames = df.index[df['SOC | donor_assignment']==df['CMO | donor_assignment']]
unmatched_obsnames =  df.index[(df['SOC | donor_assignment']!=df['CMO | donor_assignment'])&(df['SOC | donor_assignment']!='donor_unassigned')]
unassigned_obsnames = df.index[df['SOC | status']=='unassigned']
doublet_obsnames = df.index[df['SOC | status']=='doublet']

df['match'] = np.nan
df.loc[matched_obsnames,'match'] = 'SOC-COM, matched'
df.loc[unmatched_obsnames,'match'] = 'SOC-COM, unmatched'
df.loc[unassigned_obsnames,'match'] = 'SOC status, unassigned'
df.loc[doublet_obsnames,'match'] = 'SOC status, doublet'
set(df['match'])
```

Example 44 (python):
```python
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  import sys
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/pandas/core/indexing.py:1817: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._setitem_single_column(loc, value, pi)
```

Example 45 (unknown):
```unknown
{'SOC status, doublet',
 'SOC status, unassigned',
 'SOC-COM, matched',
 'SOC-COM, unmatched'}
```

Example 46 (python):
```python
sns.violinplot(data=df, x="CMO | ratio", y="match")
plt.xlim(0.5, 4)
plt.show()

df['match'].value_counts()
```

Example 47 (javascript):
```javascript
SOC-COM, matched          40208
SOC-COM, unmatched        21454
SOC status, doublet        8103
SOC status, unassigned     3346
Name: match, dtype: int64
```

Example 48 (python):
```python
from sklearn.metrics import roc_curve
```

Example 49 (python):
```python
df_s = df[df['match'].isin(['SOC-COM, matched','SOC-COM, unmatched'])]
X = np.array(df_s['CMO | ratio']) # .reshape(-1, 1)
y = np.array(df_s['match'] == 'SOC-COM, matched')
```

Example 50 (unknown):
```unknown
fpr,tpr,thresholds = roc_curve(y, X)
fpr,tpr,thresholds
```

Example 51 (unknown):
```unknown
(array([0.        , 0.        , 0.        , ..., 0.99888133, 0.99902116,
        1.        ]),
 array([0.00000000e+00, 2.48706725e-05, 1.09430959e-03, ...,
        1.00000000e+00, 1.00000000e+00, 1.00000000e+00]),
 array([15.5164995, 14.5164995,  8.282574 , ...,  1.0002261,  1.0001545,
         1.       ]))
```

Example 52 (python):
```python
optimal_idx = np.argmax(tpr - fpr)
print(f'optimal threshold to make the "tpr - fpr" maximum: {thresholds[optimal_idx]}')
print(f'true positive rate with the threshold: {tpr[optimal_idx]}')
print(f'false positive rate with the threshold: {fpr[optimal_idx]}')
```

Example 53 (unknown):
```unknown
optimal threshold to make the "tpr - fpr" maximum: 1.4452686
true positive rate with the threshold: 0.5862763629128531
false positive rate with the threshold: 0.2780367297473665
```

Example 54 (python):
```python
sns.violinplot(data=df_s, x="CMO | ratio", y="match")
plt.axvline(thresholds[optimal_idx], color='red', linestyle='dashed', linewidth=1)
plt.xlim(0.5, 4)
plt.show()
# red dashed line is the threshold which gives maximum "tpr - fpr" value
```

Example 55 (python):
```python
optimal_idx = sum(fpr < 0.05)-1
print(f'optimal threshold to make the fpr lower than 0.05: {thresholds[optimal_idx]}')
print(f'true positive rate with the threshold: {tpr[optimal_idx]}')
print(f'false positive rate with the threshold: {fpr[optimal_idx]}')
```

Example 56 (unknown):
```unknown
optimal threshold to make the fpr lower than 0.05: 2.4282365
true positive rate with the threshold: 0.23104854755272583
false positive rate with the threshold: 0.049967372051831827
```

Example 57 (python):
```python
sns.violinplot(data=df_s, x="CMO | ratio", y="match")
plt.axvline(thresholds[optimal_idx], color='red', linestyle='dashed', linewidth=1)
plt.xlim(0.5, 4)
plt.show()
# red dashed line is the threshold which gives maximum "tpr - fpr" value
```

Example 58 (unknown):
```unknown
df = df['combinedID']==''
```

Example 59 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]
# select only SOC | donor-assigned ones
df = df[df['SOC | donor_assignment'] != 'donor_unassigned']
print(df.shape)

donors = ['Hst36','Hst39','Hst40']
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']
# set order
df['SOC | donor_assignment'] = df['SOC | donor_assignment'].astype('category')
df['SOC | donor_assignment'].cat.reorder_categories(donors, inplace=True)

fig,axs = plt.subplots(nrows=3, ncols=3, figsize=(17,10)) 
for sam in samples:
    df_s = df[df['combinedID']==sam]
    
    for don in donors:
        row_no = samples.index(sam)
        col_no = donors.index(don)
        
        sns.violinplot(data=df_s, 
                       x=f'CMO | {don}',
                       y=f'SOC | donor_assignment',
                       ax=axs[row_no,col_no])
        axs[row_no,col_no].set_xlabel(f'{don} tag count')
        axs[row_no,col_no].set_title(sam)
        axs[row_no,col_no].set_xlim(-1000, 30000)

plt.tight_layout(h_pad=3.0)
```

Example 60 (unknown):
```unknown
(61662, 35)
```

Example 61 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]
# select only SOC | donor-assigned ones
df = df[df['SOC | donor_assignment'] != 'donor_unassigned']

# remove double-assigned nuclei
# df = df[[',' not in x for x in df['CMO | donor_assignment']]]
# print(df.shape)

donors = ['Hst36','Hst39','Hst40']
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']

# set color
df['color'] = 'grey'
df.loc[df.index[df['SOC | donor_assignment']=='Hst39'],'color'] = 'blue'
df.loc[df.index[df['SOC | donor_assignment']=='Hst40'],'color'] = 'red'

# set order
# df['CMO | donor_assignment'] = df['CMO | donor_assignment'].astype('category')
# df['CMO | donor_assignment'].cat.reorder_categories(donors, inplace=True)

fig,axs = plt.subplots(nrows=1, ncols=3, figsize=(20,6)) 

for sam in samples:
    # subset each sample
    df_s = df[df['combinedID']==sam]
    
    for don in donors:
        row_no = samples.index(sam)
        # col_no = donors.index(don)
        
        # subset each donor (assume SOC is giving the correct donor)
        # df_ss = df_s[df_s['SOC | donor_assignment']==don]
        
        axs[row_no].scatter( 
                       x=np.log10(df_s['CMO | Hst39']),
                       y=np.log10(df_s['CMO | Hst40']),
                       # x=df_s['CMO | Hst39'],
                       # y=df_s['CMO | Hst40'],
                       c=df_s['color'],
                        s=2
                       )
        axs[row_no].set_xlabel('CMO | Hst39 count')
        axs[row_no].set_ylabel('CMO | Hst40 count')
        axs[row_no].set_title(f'{sam}, SOC|{don}')
        # axs[row_no,col_no].set_xlim(-1000, 50000)

plt.tight_layout(w_pad=5.0, h_pad=3.0)
```

Example 62 (python):
```python
np.mean(df['CMO | Hst39'])
```

Example 63 (unknown):
```unknown
3688.686516817489
```

Example 64 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]
# select only SOC | donor-assigned ones
df = df[df['SOC | donor_assignment'] != 'donor_unassigned']

# remove double-assigned nuclei
df = df[[',' not in x for x in df['CMO | donor_assignment']]]
print(df.shape)

donors = ['Hst36','Hst39','Hst40']
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']

# set order
df['CMO | donor_assignment'] = df['CMO | donor_assignment'].astype('category')
df['CMO | donor_assignment'].cat.reorder_categories(donors, inplace=True)

fig,axs = plt.subplots(nrows=3, ncols=3, figsize=(20,17)) 

for sam in samples:
    # subset each sample
    df_s = df[df['combinedID']==sam]
    
    for don in donors:
        row_no = samples.index(sam)
        col_no = donors.index(don)
        
        # subset each donor (assume SOC is giving the correct donor)
        df_ss = df_s[df_s['SOC | donor_assignment']==don]
        
        if len(df_ss)==0:
            continue
        
        axs[row_no,col_no].scatter(data=df_ss, 
                       x='CMO | Hst39',
                       y='CMO | Hst40',
                       )
        axs[row_no,col_no].set_xlabel('CMO | Hst39 count')
        axs[row_no,col_no].set_ylabel('CMO | Hst40 count')
        axs[row_no,col_no].set_title(f'{sam}, SOC|{don}')
        axs[row_no,col_no].set_xlim(-1000, 50000)

plt.tight_layout(w_pad=5.0, h_pad=3.0)
```

Example 65 (unknown):
```unknown
(61641, 37)
```

Example 66 (python):
```python
samplefile = pd.read_csv('/nfs/team205/heart/mapped/CMO/csv/multiome-foetal_20221221.csv')
samplefile.set_index('CMO_ID',inplace=True)
samplefile
```

Example 67 (python):
```python
cmo_ID = '7089STDY13216916'
cmo_raw = sc.read_10x_mtx(f'/nfs/team205/heart/mapped/CMO/{cmo_ID}/raw_feature_bc_matrix',
                     gex_only=False)
cmo_raw.var_names = [x.split('-')[0] for x in cmo_raw.var_names]


# get multiome ID
multi_ID = samplefile.loc[cmo_ID,'Multiome_ID']

# modify obs_names
cmo_raw.obs_names = multi_ID + '_' + cmo_raw.obs_names

# label
cmo_raw.obs['label'] = 'not-cell'
shared = set(adata.obs_names).intersection(cmo_raw.obs_names)
cmo_raw.obs.loc[shared,'label'] = adata.obs.loc[shared, 'SOC | donor_assignment']

# color
cmo_raw.obs['color'] = cmo_raw.obs['label'].copy()
cmo_raw.obs.replace({'color':{
    'Hst36':'green',
    'Hst39':'blue',
    'Hst40':'red',
    'donor_unassigned':'grey',
    'not-cell':'grey'
}},inplace=True)

tag = 'Hst40'
df = cmo_raw.to_df()[[tag]]
df.sort_values(tag,ascending=False,inplace=True)
df['rank'] = np.array(range(len(df)))+1

plt.rcParams['figure.figsize'] = 6,4
plt.scatter(x=df['rank'], y=np.log10(df[tag]), c=cmo_raw.obs['color'].reindex(df.index), s=5)
# plt.scatter(x=df['rank'], y=df[tag], c=cmo_raw.obs['color'].reindex(df.index), s=5)
plt.xscale('log')
plt.xlabel('Droplet ID ranked by count')
plt.ylabel(f'CMO count per droplet (log10)')
plt.title(tag)
plt.show()
plt.clf()
```

Example 68 (unknown):
```unknown
<Figure size 432x288 with 0 Axes>
```

Example 69 (python):
```python
cmo_ID = '7089STDY13216916'
cmo_raw = sc.read_10x_mtx(f'/nfs/team205/heart/mapped/CMO/{cmo_ID}/raw_feature_bc_matrix',
                     gex_only=False)
cmo_raw.var_names = [x.split('-')[0] for x in cmo_raw.var_names]


# get multiome ID
multi_ID = samplefile.loc[cmo_ID,'Multiome_ID']

# modify obs_names
cmo_raw.obs_names = multi_ID + '_' + cmo_raw.obs_names

# label
cmo_raw.obs['label'] = 'not-cell'
shared = set(adata.obs_names).intersection(cmo_raw.obs_names)
cmo_raw.obs.loc[shared,'label'] = adata.obs.loc[shared, 'SOC | donor_assignment']

# color
cmo_raw.obs['color'] = cmo_raw.obs['label'].copy()
cmo_raw.obs.replace({'color':{
    'Hst36':'green',
    'Hst39':'blue',
    'Hst40':'red',
    'donor_unassigned':'grey',
    'not-cell':'grey'
}},inplace=True)

tags = ['Hst39 (tag6)', 'Hst40 (tag4)']

fig,axs = plt.subplots(nrows=1, ncols=2, figsize=(12,5))
fig.suptitle(multi_ID)
for i,tg in enumerate(tags):
    don = tg.split(' (')[0]
    df = cmo_raw.to_df()[[don]]
    df.sort_values(don,ascending=False,inplace=True)
    df['rank'] = np.array(range(len(df)))+1

    axs[i].scatter(x=df['rank'], y=np.log10(df[don]), c=cmo_raw.obs['color'].reindex(df.index), s=10)
    # axs[i].scatter(x=df['rank'], y=df[don], c=cmo_raw.obs['color'].reindex(df.index), s=1)
    axs[i].set_xscale('log')
    axs[i].set_xlabel('Droplets ranked by count')
    axs[i].set_ylabel(f'CMO count per droplet (log10)')
    axs[i].set_title(tg)

plt.tight_layout(w_pad=3.0)
```

Example 70 (unknown):
```unknown
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/pandas/core/arraylike.py:364: RuntimeWarning: divide by zero encountered in log10
  result = getattr(ufunc, method)(*inputs, **kwargs)
```

Example 71 (unknown):
```unknown
cmo_raw.to_df()
```

Example 72 (python):
```python
cmo_ID = '7089STDY13216917'
cmo_raw = sc.read_10x_mtx(f'/nfs/team205/heart/mapped/CMO/{cmo_ID}/raw_feature_bc_matrix',
                     gex_only=False)
cmo_raw.var_names = [x.split('-')[0] for x in cmo_raw.var_names]


# get multiome ID
multi_ID = samplefile.loc[cmo_ID,'Multiome_ID']

# modify obs_names
cmo_raw.obs_names = multi_ID + '_' + cmo_raw.obs_names

# label
cmo_raw.obs['label'] = 'not-cell'
shared = set(adata.obs_names).intersection(cmo_raw.obs_names)
cmo_raw.obs.loc[shared,'label'] = adata.obs.loc[shared, 'SOC | donor_assignment']

# color
cmo_raw.obs['color'] = cmo_raw.obs['label'].copy()
cmo_raw.obs.replace({'color':{
    'Hst36':'green',
    'Hst39':'blue',
    'Hst40':'red',
    'donor_unassigned':'white',
    'not-cell':'grey'
}},inplace=True)

tags = ['Hst39 (tag6)', 'Hst40 (tag4)']

fig,axs = plt.subplots(nrows=1, ncols=2, figsize=(12,5))
fig.suptitle(multi_ID)
for i,tg in enumerate(tags):
    don = tg.split(' (')[0]
    df = cmo_raw.to_df()[[don]]
    df.sort_values(don,ascending=False,inplace=True)
    df['rank'] = np.array(range(len(df)))+1

    axs[i].scatter(x=df['rank'], y=np.log10(df[don]), c=cmo_raw.obs['color'].reindex(df.index), s=10)
    # axs[i].scatter(x=df['rank'], y=df[don], c=cmo_raw.obs['color'].reindex(df.index), s=1)
    axs[i].set_xscale('log')
    axs[i].set_xlabel('Droplets ranked by count')
    axs[i].set_ylabel(f'CMO count per droplet (log10)')
    axs[i].set_title(tg)

plt.tight_layout(w_pad=3.0)
```

Example 73 (python):
```python
cmo_ID = '7089STDY13216918'
cmo_raw = sc.read_10x_mtx(f'/nfs/team205/heart/mapped/CMO/{cmo_ID}/raw_feature_bc_matrix',
                     gex_only=False)
cmo_raw.var_names = [x.split('-')[0] for x in cmo_raw.var_names]


# get multiome ID
multi_ID = samplefile.loc[cmo_ID,'Multiome_ID']

# modify obs_names
cmo_raw.obs_names = multi_ID + '_' + cmo_raw.obs_names

# label
cmo_raw.obs['label'] = 'not-cell'
shared = set(adata.obs_names).intersection(cmo_raw.obs_names)
cmo_raw.obs.loc[shared,'label'] = adata.obs.loc[shared, 'SOC | donor_assignment']

# color
cmo_raw.obs['color'] = cmo_raw.obs['label'].copy()
cmo_raw.obs.replace({'color':{
    'Hst36':'green',
    'Hst39':'blue',
    'Hst40':'red',
    'donor_unassigned':'grey',
    'not-cell':'grey'
}},inplace=True)

tags = ['Hst36 (tag9)', 'Hst39 (tag8)', 'Hst40 (tag5)']

fig,axs = plt.subplots(nrows=1, ncols=3, figsize=(17,5))
fig.suptitle(multi_ID)
for i,tg in enumerate(tags):
    don = tg.split(' (')[0]
    df = cmo_raw.to_df()[[don]]
    df.sort_values(don,ascending=False,inplace=True)
    df['rank'] = np.array(range(len(df)))+1

    axs[i].scatter(x=df['rank'], y=np.log10(df[don]), c=cmo_raw.obs['color'].reindex(df.index), s=10)
    # axs[i].scatter(x=df['rank'], y=df[don], c=cmo_raw.obs['color'].reindex(df.index), s=1)
    axs[i].set_xscale('log')
    axs[i].set_xlabel('Droplets ranked by count')
    axs[i].set_ylabel(f'CMO count per droplet (log10)')
    axs[i].set_title(tg)

plt.tight_layout(w_pad=3.0)
```

Example 74 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]
# select only SOC | donor-assigned ones
df = df[df['SOC | donor_assignment'] != 'donor_unassigned']

# remove double-assigned nuclei
df = df[[',' not in x for x in df['CMO | donor_assignment']]]
print(df.shape)

donors = ['Hst36','Hst39','Hst40']
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']

# set order
df['CMO | donor_assignment'] = df['CMO | donor_assignment'].astype('category')
df['CMO | donor_assignment'].cat.reorder_categories(donors, inplace=True)

fig,axs = plt.subplots(nrows=3, ncols=3, figsize=(20,10)) 

for sam in samples:
    # subset each sample
    df_s = df[df['combinedID']==sam]
    
    for don in donors:
        row_no = samples.index(sam)
        col_no = donors.index(don)
        
        # subset each donor (assume SOC is giving the correct donor)
        df_ss = df_s[df_s['SOC | donor_assignment']==don]
        
        if len(df_ss)==0:
            continue
        
        sns.violinplot(data=df_ss, 
                       x=f'CMO | top_count',
                       y=f'CMO | donor_assignment',
                       ax=axs[row_no,col_no])
        axs[row_no,col_no].set_xlabel(f'top_count')
        axs[row_no,col_no].set_title(f'{sam}, SOC|{don}')
        axs[row_no,col_no].set_xlim(-1000, 50000)

plt.tight_layout(w_pad=5.0, h_pad=3.0)
```

Example 75 (unknown):
```unknown
(61641, 37)
```

Example 76 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]
# select only SOC | donor-assigned ones
df = df[df['SOC | donor_assignment'] != 'donor_unassigned']

# remove double-assigned nuclei
df = df[[',' not in x for x in df['CMO | donor_assignment']]]
print(df.shape)

donors = ['Hst36','Hst39','Hst40']
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']

# set order
df['CMO | donor_assignment'] = df['CMO | donor_assignment'].astype('category')
df['CMO | donor_assignment'].cat.reorder_categories(donors, inplace=True)

fig,axs = plt.subplots(nrows=3, ncols=3, figsize=(20,10)) 

for sam in samples:
    # subset each sample
    df_s = df[df['combinedID']==sam]
    
    for don in donors:
        row_no = samples.index(sam)
        col_no = donors.index(don)
        
        # subset each donor (assume SOC is giving the correct donor)
        df_ss = df_s[df_s['SOC | donor_assignment']==don]
        
        if len(df_ss)==0:
            continue
        
        sns.violinplot(data=df_ss, 
                       x=f'CMO | delta',
                       y=f'CMO | donor_assignment',
                       ax=axs[row_no,col_no])
        axs[row_no,col_no].set_xlabel(f'delta')
        axs[row_no,col_no].set_title(f'{sam}, SOC|{don}')
        axs[row_no,col_no].set_xlim(-5000, 30000)

plt.tight_layout(w_pad=5.0, h_pad=3.0)
```

Example 77 (unknown):
```unknown
(61641, 36)
```

Example 78 (python):
```python
np.mean(df['CMO | Hst39'])
```

Example 79 (unknown):
```unknown
3688.8545935335246
```

Example 80 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]
# select only SOC | donor-assigned ones
df = df[df['SOC | donor_assignment'] != 'donor_unassigned']

# remove double-assigned nuclei
df = df[[',' not in x for x in df['CMO | donor_assignment']]]
print(df.shape)

donors = ['Hst36','Hst39','Hst40']
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']

# set order
df['CMO | donor_assignment'] = df['CMO | donor_assignment'].astype('category')
df['CMO | donor_assignment'].cat.reorder_categories(donors, inplace=True)

fig,axs = plt.subplots(nrows=3, ncols=3, figsize=(20,10)) 

for sam in samples:
    # subset each sample
    df_s = df[df['combinedID']==sam]
    
    for don in donors:
        row_no = samples.index(sam)
        col_no = donors.index(don)
        
        # subset each donor (assume SOC is giving the correct donor)
        df_ss = df_s[df_s['SOC | donor_assignment']==don]
        
        if len(df_ss)==0:
            continue
        
        sns.violinplot(data=df_ss, 
                       x=f'CMO | prop_delta',
                       y=f'CMO | donor_assignment',
                       ax=axs[row_no,col_no])
        axs[row_no,col_no].set_xlabel(f'prop_delta')
        axs[row_no,col_no].set_title(f'{sam}, SOC|{don}')

plt.tight_layout(w_pad=5.0, h_pad=3.0)
```

Example 81 (unknown):
```unknown
(61641, 35)
```

Example 82 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]
# select only SOC | donor-assigned ones
df = df[df['SOC | donor_assignment'] != 'donor_unassigned']

# remove double-assigned nuclei
df = df[[',' not in x for x in df['CMO | donor_assignment']]]
print(df.shape)

donors = ['Hst36','Hst39','Hst40']
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']

# set order
df['CMO | donor_assignment'] = df['CMO | donor_assignment'].astype('category')
df['CMO | donor_assignment'].cat.reorder_categories(donors, inplace=True)

fig,axs = plt.subplots(nrows=3, ncols=3, figsize=(20,10)) 

for sam in samples:
    # subset each sample
    df_s = df[df['combinedID']==sam]
    
    for don in donors:
        row_no = samples.index(sam)
        col_no = donors.index(don)
        
        # subset each donor (assume SOC is giving the correct donor)
        df_ss = df_s[df_s['SOC | donor_assignment']==don]
        
        if len(df_ss)==0:
            continue
        
        sns.violinplot(data=df_ss, 
                       x=f'CMO | prop_delta',
                       y=f'CMO | donor_assignment',
                       ax=axs[row_no,col_no])
        axs[row_no,col_no].set_xlabel(f'prop_delta')
        axs[row_no,col_no].set_title(f'{sam}, SOC|{don}')

plt.tight_layout(w_pad=5.0, h_pad=3.0)
```

Example 83 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]
# select only SOC | donor-assigned ones
df = df[df['SOC | donor_assignment'] != 'donor_unassigned']
print(df.shape)

donors = ['Hst36','Hst39','Hst40']
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']

# label correct one and uncorrect



df['match'] = 

fig,axs = plt.subplots(nrows=3, ncols=3, figsize=(17,10)) 
for sam in samples:
    # subset each sample
    df_s = df[df['combinedID']==sam]
    
    # label correct one and uncorrect
    df
    
    for don in donors:
        row_no = samples.index(sam)
        col_no = donors.index(don)
        
        sns.violinplot(data=df_s, 
                       x=f'CMO | {don}',
                       y=f'SOC | donor_assignment',
                       ax=axs[row_no,col_no])
        axs[row_no,col_no].set_xlabel(f'{don} tag count')
        axs[row_no,col_no].set_title(sam)
        axs[row_no,col_no].set_xlim(-1000, 30000)

plt.tight_layout(h_pad=3.0)
```

Example 84 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]
print(df.shape)

donors = ['Hst36','Hst39','Hst40']
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']

fig,axs = plt.subplots(nrows=3, ncols=3, figsize=(20,10)) 

for sam in samples:
    df_s = df[df['combinedID']==sam]
    
    for don in donors:
        row_no = samples.index(sam)
        col_no = donors.index(don)
        
        sns.violinplot(data=df_s, 
                       x=f'CMO | {don}',
                       y=f'CMO | donor_assignment',
                       ax=axs[row_no,col_no])
        axs[row_no,col_no].set_xlabel(f'{don} tag count')
        axs[row_no,col_no].set_title(sam)
        axs[row_no,col_no].set_xlim(-1000, 30000)

plt.tight_layout(h_pad=3.0)
```

Example 85 (unknown):
```unknown
(73111, 35)
```

Example 86 (python):
```python
# multiplexed samples apart from '7089STDY13216923_BHF_F_Hea13242530'(no SOC assignment)
df = adata.obs[adata.obs['combinedID'].isin(['7089STDY13216920_BHF_F_Hea13242527',
                                             '7089STDY13216921_BHF_F_Hea13242528',
                                             '7089STDY13216922_BHF_F_Hea13242529',])]
print(df.shape)

donors = ['Hst36','Hst39','Hst40']
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']

for sam in samples:
    df_s = df[df['combinedID']==sam]
        
    sns.violinplot(data=df_s, 
                   x=f'CMO | prop_delta',
                   y=f'SOC | donor_assignment')
    plt.xlabel('prop_delta')
    plt.title(sam)
    plt.show()
```

Example 87 (unknown):
```unknown
(73111, 35)
```

Example 88 (unknown):
```unknown
samples = ['7089STDY13216920_BHF_F_Hea13242527',
           '7089STDY13216921_BHF_F_Hea13242528',
           '7089STDY13216922_BHF_F_Hea13242529']
samples.index('7089STDY13216921_BHF_F_Hea13242528')
```

Example 89 (unknown):
```unknown
df[df['combinedID']=='7089STDY13216920_BHF_F_Hea13242527']
```

Example 90 (unknown):
```unknown
df['combinedID']
```

Example 91 (unknown):
```unknown
{'7089STDY13216920_BHF_F_Hea13242527',
 '7089STDY13216921_BHF_F_Hea13242528',
 '7089STDY13216922_BHF_F_Hea13242529',
 '7089STDY13216923_BHF_F_Hea13242530'}
```

Example 92 (python):
```python
df = adata.obs[adata.obs['combinedID'].isin(multiplexed_samples)]
df['combinedID'] = df['combinedID'].astype('str')
```

Example 93 (python):
```python
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
```

Example 94 (python):
```python
donor = 'Hst36'
sns.violinplot(data=df, x=f'CMO | {donor}', y="combinedID", vmax=5000)
plt.title(f'{donor} tag count')
plt.show()
```

Example 95 (python):
```python
donor = 'Hst36'
sns.violinplot(data=df, x=f'CMO | {donor}', y="combinedID", vmax=5000)
plt.xlim(-1000, 30000)
plt.title(f'{donor} tag count')
plt.show()

donor = 'Hst39'
sns.violinplot(data=df, x=f'CMO | {donor}', y="combinedID", vmax=5000)
plt.xlim(-1000, 30000)
plt.title(f'{donor} tag count')
plt.show()

donor = 'Hst40'
sns.violinplot(data=df, x=f'CMO | {donor}', y="combinedID", vmax=5000)
plt.xlim(-1000, 30000)
plt.title(f'{donor} tag count')
plt.show()
```

Example 96 (unknown):
```unknown
multiplexed_samples
```

Example 97 (unknown):
```unknown
['7089STDY13216920_BHF_F_Hea13242527',
 '7089STDY13216921_BHF_F_Hea13242528',
 '7089STDY13216922_BHF_F_Hea13242529',
 '7089STDY13216923_BHF_F_Hea13242530']
```

Example 98 (python):
```python
pd.crosstab(df['SOC | donor_assignment'],df['CMO | donor_assignment'])
```

Example 99 (python):
```python
SOC_39_CMO_39 = df.index[(df['SOC | donor_assignment']=='Hst39')&(df['CMO | donor_assignment']=='Hst39')]
SOC_39_CMO_40 = df.index[(df['SOC | donor_assignment']=='Hst39')&(df['CMO | donor_assignment']=='Hst40')]
SOC_40_CMO_40 = df.index[(df['SOC | donor_assignment']=='Hst40')&(df['CMO | donor_assignment']=='Hst40')]
SOC_40_CMO_39 = df.index[(df['SOC | donor_assignment']=='Hst40')&(df['CMO | donor_assignment']=='Hst39')]

df['match_39-40'] = np.nan
df.loc[SOC_39_CMO_39,'match_39-40'] = 'SOC-39_CMO-39'
df.loc[SOC_39_CMO_40,'match_39-40'] = 'SOC-39_CMO-40'
df.loc[SOC_40_CMO_40,'match_39-40'] = 'SOC-40_CMO-40'
df.loc[SOC_40_CMO_39,'match_39-40'] = 'SOC-40_CMO-39'
```

Example 100 (python):
```python
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/ipykernel_launcher.py:6: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/pandas/core/indexing.py:1817: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._setitem_single_column(loc, value, pi)
```

Example 101 (unknown):
```unknown
df['match_39-40'] = df['match_39-40'].astype('category')
df['match_39-40'].cat.reorder_categories([
    'SOC-39_CMO-39',
    'SOC-39_CMO-40',
    'SOC-40_CMO-40',
    'SOC-40_CMO-39'
], inplace=True)
```

Example 102 (python):
```python
/home/jovyan/my-conda-envs/cellpymc/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.
```

Example 103 (python):
```python
for sam in ['7089STDY13216920_BHF_F_Hea13242527',
 '7089STDY13216921_BHF_F_Hea13242528',
 '7089STDY13216922_BHF_F_Hea13242529']:
    df_s = df[df['combinedID']==sam]
    sns.violinplot(data=df_s, x='CMO | Hst39', y="match_39-40", vmax=5000)
    plt.xlim(-2000, 30000)
    plt.title(sam)
    plt.show()
```

Example 104 (python):
```python
for sam in ['7089STDY13216920_BHF_F_Hea13242527',
 '7089STDY13216921_BHF_F_Hea13242528',
 '7089STDY13216922_BHF_F_Hea13242529']:
    df_s = df[df['combinedID']==sam]
    sns.violinplot(data=df_s, x='CMO | Hst40', y="match_39-40", vmax=5000)
    plt.xlim(-2000, 30000)
    plt.title(sam)
    plt.show()
```

Example 105 (python):
```python
for sam in ['7089STDY13216920_BHF_F_Hea13242527',
 '7089STDY13216921_BHF_F_Hea13242528',
 '7089STDY13216922_BHF_F_Hea13242529']:
    df_s = df[df['combinedID']==sam]
    sns.violinplot(data=df_s, x='CMO | Hst39', y="match", vmax=5000)
    plt.xlim(-1000, 20000)
    plt.title(sam)
    plt.show()
```

Example 106 (python):
```python
df - cmo

for don in set(df['CMO | donor_assignment']):
    
    if ',' not in don:
    
        df_s = df[df['CMO | donor_assignment']==don]
        plt.scatter(df_s['CMO | prop_delta'],df_s[f'CMO | {don}'])
        plt.title(f'nuclei which had max-count={don} tag')
        plt.xlabel('prop_delta')
        plt.ylabel(f'{don} count')
        plt.show()
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/scnRNA_Integration-Annotation/T21/T21HeartsFurtherQC.html

**Contents:**
- Investigate Immune cluster¶
- Investigate epicardium¶
- Investigate endothelium¶
- Investigate neural¶
- Investigate mesenchymal¶
- Investigate cardiomyocytes¶
- Investigate the overall object with the other cells removed¶
- Process the dataset with all contaminating cells removed, add metadata information¶

**Examples:**

Example 1 (python):
```python
# Import packages, initial settings

import numpy as np
import matplotlib.pyplot as plt
import scanpy as sc
import pandas as pd
import anndata
import scrublet as scr
import seaborn as sns
import magic
import pybiomart
import six
import sys
sys.modules['sklearn.externals.six'] = six
import scanorama
import harmonypy
import bbknn
import gseapy
from gseapy.plot import barplot, dotplot
import scvelo as scv
import scvi

plt.rcParams['pdf.fonttype'] = 42
plt.rcParams['ps.fonttype'] = 42

sc.settings.verbosity = 3  # verbosity: errors (0), warnings (1), info (2), hints (3)

from matplotlib.colors import LinearSegmentedColormap
cmap = LinearSegmentedColormap.from_list(name='gene_cmap', colors=['lightgrey', 'thistle', 'red', 'darkred']) 

sc.settings.set_figure_params(dpi=80, color_map='viridis', vector_friendly=False,  dpi_save=300, facecolor='white')
sc.logging.print_version_and_date()
#sc.logging.print_versions()
```

Example 2 (python):
```python
adata=sc.read('./Objects/T21_multiome_RNA_foetal.h5ad')
print(adata.shape)
```

Example 3 (python):
```python
# set gene name as var_names 
adata.var['gene_id']=adata.var_names.copy()
adata.var.set_index('gene_name',inplace=True)
adata.var_names=adata.var_names.astype('str')
adata.var_names_make_unique()
```

Example 4 (python):
```python
cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP','RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2','USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67','TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1','CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF','NEK2','G2E3','GAS2L3','CBX5','CENPA']
s_genes = cell_cycle_genes[:43]
s_genes = [x for x in s_genes if x in adata.var_names]
print('S genes:', s_genes, '\n')
g2m_genes = cell_cycle_genes[43:]
g2m_genes = [x for x in g2m_genes if x in adata.var_names]
print('G2 genes:', g2m_genes, '\n')
cell_cycle_genes = [x for x in cell_cycle_genes if x in adata.var_names]

hb_genes1 = ['HBB','HBD','HBG1','HBG2','HBE1','HBZ','HBM','HBA2','HBA1', 'HBQ1']
hb_genes1 = [x for x in hb_genes1 if x in adata.var_names]
print('Hemoglobin genes set 1:', hb_genes1, '\n')

hb_genes2 = ['AHSP','GYPA','RHAG','GFI1B','KEL','EPB42','ALAS2','TSPO2','GYPB','GYPE','SPTA1','RHCE','KLF1','RHD','SLC4A1','SNX22','SQSTM1','SOD1','FTH1','MAP1LC3B','SLC25A37','TRIM58','HMBS']
hb_genes2 = [x for x in hb_genes2 if x in adata.var_names]
print('Hemoglobin genes set 2:', hb_genes2, '\n')

mito_genes=['MT-ND1','MT-ND2','MT-CO1','MT-CO2','MT-ATP8','MT-ATP6','MT-CO3','MT-ND3','MT-ND4L', 'MT-ND4', 'MT-ND5', 'MT-ND6', 'MT-CYB']
mito_genes = [x for x in mito_genes if x in adata.var_names]
print('Mitochondrial genes:', mito_genes, '\n')

ribo_genes = ['RPLP0','RPLP1','RPLP2','RPL3','RPL3L','RPL4','RPL5','RPL6','RPL7','RPL7A','RPL7L1','RPL8','RPL9','RPL10','RPL10A','RPL10L','RPL11','RPL12','RPL13','RPL13A','RPL14','RPL15','RPL17','RPL18','RPL18A','RPL19','RPL21','RPL22','RPL22L1','RPL23','RPL23A','RPL24','RPL26','RPL26L1','RPL27','RPL27A','RPL28','RPL29','RPL30','RPL31','RPL32','RPL34','RPL35','RPL35A','RPL36','RPL36A','RPL36AL','RPL37','RPL37A','RPL38','RPL39','RPL39L','UBA52','RPL41','RPSA','RPS2','RPS3','RPS3A','RPS4X','RPS4Y1','RPS4Y2','RPS5','RPS6','RPS7','RPS8','RPS9','RPS10','RPS11','RPS12','RPS13','RPS14','RPS15','RPS15A','RPS16','RPS17','RPS18','RPS19','RPS20','RPS21','RPS23','RPS24','RPS25','RPS26','RPS27','RPS27A','RPS27L','RPS28','RPS29','FAU']
ribo_genes = [x for x in ribo_genes if x in adata.var_names]
print('Ribo genes:', ribo_genes, '\n')

stress_genes=['FOS','FOSB','JUN','JUNB','JUND','ZFP36','SOCS3','CEBPD','CEBPB','ATF3','EGR1','HSPA1A','HSPA1B','HSP90AA1','HSP90AB1','HSPA8','HSPB1','HSPE1','IER3','IER2','BTG1','BTG2','DUSP1']
stress_genes = [x for x in stress_genes if x in adata.var_names]
print('Tissue dissociation stress associated genes:', stress_genes, '\n')

FoxProteins=['FOXA1','FOXA2','FOXA3','FOXB1','FOXB2','FOXC1','FOXC2','FOXD1','FOXD2','FOXD3','FOXD4','FOXD4L1','FOXD4L3','FOXD4L4','FOXD4L5','FOXD4L6','FOXE1','FOXE3','FOXF1','FOXF2','FOXG1','FOXH1','FOXI1','FOXI2','FOXI3','FOXJ1','FOXJ2','FOXJ3','FOXK1','FOXK2','FOXL1','FOXL2','FOXM1','FOXN1','FOXN2','FOXN3','FOXO1','FOXO3','FOXO4','FOXO6','FOXP1','FOXP2','FOXP3','FOXP4','FOXQ1','FOXR1','FOXR2','FOXS1']
chrY_genes=['SRY', 'RPS4Y1', 'ZFY', 'ZFY-AS1', 'LINC00278', 'PCDH11Y', 'TTTY2B', 'TBL1Y', 'PRKY', 'USP9Y', 'DDX3Y', 'UTY', 'TMSB4Y', 'NLGN4Y', 'TTTY14', 'KDM5D', 'TTTY10', 'EIF1AY', 'PRORY', 'LINC00266-4P']
MembraneProteins1=['AADAT','AAK1','ABCA12','ABCA7','ABCB1','ABCB11','ABCB4','ABCB6','ABCC1','ABCC3','ABCC4','ABCC5','ABCC6','ABCG2','ABHD3','ABI1','ABLIM3','AC004691.2','AC004922.1','AC005324.3','AC005726.1','AC008403.1','AC138696.1','ACIN1','ACKR3','ACLY','ACOT13','ACOT9','ACTN1','ACTR10','ACTR6','ADA','ADAM10','ADAM12','ADAM22','ADAM28','ADAM33','ADAMTS1','ADAMTS2','ADAMTSL4','ADAP1','ADCK5','ADD1','ADD2','ADGRB2','ADGRD1','ADGRG2','ADGRG7','ADGRL2','ADH1A','ADH1B','ADH1C','ADH7','ADK','ADORA1','ADRA1B','ADRM1','ADSS','AFAP1L2','AFDN','AFG1L','AGER','AGRN','AHNAK','AHNAK2','AIFM2','AKAP10','AKAP11','AKAP12','AKR1B10','AKR1B15','AL132780.3','AL139300.1','AL355315.1','AL358113.1','AL645922.1','ALDH3A1','ALK','ALOX15B','ALOXE3','ALPI','ALPP','ALPPL2','ALS2CR12','AMACR','AMDHD1','AMER1','AMER2','AMIGO2','AMOT','AMOTL1','AMOTL2','AMPH','ANK2','ANK3','ANKEF1','ANKFN1','ANKIB1','ANKLE2','ANKRA2','ANKRD13A','ANKRD35','ANKRD42','ANKRD45','ANKS1B','ANO1','ANO10','ANO2','ANO6','ANPEP','ANXA1','ANXA13','ANXA2','ANXA3','AP2M1','APBB1','APBB1IP','APC','APLP1','APMAP','APOC3','APOD','APOLD1','AQP1','AQP10','AQP3','AQP4','AQP5','ARAP1','ARF1','ARHGAP18','ARHGAP19','ARHGAP21','ARHGAP22','ARHGAP23','ARHGAP24','ARHGAP28','ARHGAP29','ARHGAP33','ARHGAP36','ARHGAP40','ARHGAP9','ARHGDIG','ARHGEF1','ARHGEF11','ARHGEF12','ARHGEF15','ARHGEF26','ARHGEF28','ARHGEF40','ARHGEF5','ARID2','ARID4A','ARID5B','ARL13A','ARL15','ARL4C','ARMC12','ARMCX5','ARPC1A','ARPC5','ARRB2','ARRDC4','ARVCF','ASAP1','ASB5','ASGR1','ASGR2','ASH2L','ASIC1','ASIC4','ASPM','ASPSCR1','ATG13','ATG3','ATG7','ATIC','ATOX1','ATP10D','ATP12A','ATP13A3','ATP1B3','ATP23','ATP2B1','ATP2B4','ATP2C2','ATP6V0A2','ATP6V1H','ATP8A2','ATP8B1','ATXN10','ATXN3','ATXN7L3','AVEN','AVPI1','AXIN2','AXL','B2M','B3GALT2','BACE1','BAG3','BAIAP2','BAIAP2L1','BANK1','BASP1','BBOF1','BCAR1','BCAR3','BCL2L12','BCORL1','BCR','BDKRB2','BEND6','BFSP1','BFSP2','BICD2','BIN2','BLNK','BLVRB','BMPR2','BMX','BOC','BOLL','BORCS5','BORCS8-MEF2B','BRK1','BRPF1','BRSK1','BSDC1','BST2','BTK','BTN2A1','BTRC','BUB1B-PAK6','BVES','BZW2','C10orf90','C11orf42','C11orf52','C12orf73','C12orf76','C14orf37','C14orf93','C16orf74','C17orf58','C17orf62','C18orf21','C19orf33','C19orf38','C19orf48','C19orf71','C1orf116','C1orf189','C1orf35','C1orf54','C1orf68','C1orf74','C1QBP','C2orf72','C3orf14','C3orf20','C3orf38','C4orf19','C4orf22','C7orf43','C8orf33','C8orf37','C9orf153','C9orf3','CA14','CA9','CABLES2','CACNA1E','CACNG4','CADM1','CALCRL','CALD1','CALML5','CALN1','CAMK2A','CAMK2B','CAMK2D','CAMK2G','CAP2','CAPS','CASKIN2','CASP4','CAST','CAV3','CAVIN1','CAVIN2','CAVIN3','CAVIN4','CBWD1','CBWD2','CBWD3','CBWD5','CBWD6','CC2D1A','CCBE1','CCDC107','CCDC112','CCDC115','CCDC124','CCDC136','CCDC144NL','CCDC15','CCDC154','CCDC170','CCDC24','CCDC28A','CCDC59','CCDC62','CCDC66','CCDC69','CCDC8','CCDC83','CCDC85C','CCDC93','CCDC97','CCNT2','CCNYL1','CCR3','CCR4','CCT3','CD109','CD163','CD164L2','CD200R1','CD226','CD248','CD274','CD2AP','CD300A','CD300LB','CD33','CD34','CD37','CD38','CD3E','CD3G','CD4','CD44','CD46','CD47','CD48','CD79B','CD81','CD84','CD86','CD8A','CD8B','CD9','CD93','CD99L2','CDAN1','CDC42EP1','CDC42EP3','CDC42EP4','CDC42SE1','CDCA4','CDH1','CDH13','CDH15','CDH17','CDH2','CDH26','CDH3','CDH4','CDH5','CDH6','CDH8','CDIPT','CDK16','CDK5','CDK5RAP2','CDK7','CDRT15','CDV3','CEACAM5','CELSR1','CENPJ','CEP162','CEP295','CEP55','CEP57L1','CEP89','CERCAM','CFAP157','CFAP206','CFAP58','CFAP65','CFB','CFL2','CFLAR','CGN','CGNL1','CHCHD7','CHIC2','CHMP2A','CHRM3','CHRNA1','CHRNA5','CHRNB4','CHRND','CIB1','CIB3','CIP2A','CKAP5','CLCA2','CLCA4','CLCN5','CLCN6','CLDN11','CLDN18','CLDN2','CLDN23','CLDN3','CLDN4','CLDN7','CLDN9','CLEC2A','CLEC4F','CLEC5A','CLECL1','CLIC2','CLIC5','CLTB','CLUL1','CMTM4','CMTM7','CMTR2','CMYA5','CNGA1','CNKSR2','CNNM1','CNNM4','CNP','CNR1','CNTD1','COBL','COBLL1','COG3','COL16A1','COL17A1','COL4A5','COL6A2','COLEC12','COLQ','COMMD4','COMTD1','COPRS','COQ7','CORIN','CORO1B','CORO2A','CORO2B','COX6B2','CPEB3','CPPED1','CPZ','CRB3','CREB5','CRIP1','CRIP2','CRK','CRLF3','CROCC','CRTC1','CRYAB','CRYBG3','CRYL1','CSDE1','CSF1','CSF1R','CSPG4','CST6','CTNNA1','CTNNB1','CTNNBIP1','CTNND1','CTNND2','CTSF','CTSV','CTTN','CTXN1','CUEDC1','CUTA','CX3CL1','CX3CR1','CXADR','CXCR2','CYB5D2','CYBRD1','CYFIP2','CYP20A1','CYP24A1','CYTH2','CYTH4','DAAM1','DAB2','DAG1','DAGLB','DAW1','DBN1','DCAF13','DCAF16','DCBLD2','DCHS1','DCHS2','DCP2','DCT','DCTN6','DCUN1D2','DDAH1','DDB2','DDIAS','DDIT4L','DDR1','DDR2','DDX58','DDX60L','DENND6B','DFFA','DGKG','DGKH','DHDDS','DHPS','DHRS7B','DIAPH1','DIAPH3','DIO2','DIS3L','DLG1','DLG2','DLG5','DLGAP3','DLL3','DMWD','DNAAF1','DNAAF4','DNAH17','DNAJA4','DNAJB13','DNAJB4','DNAJC18','DNAJC5','DNAJC6','DNAJC9','DOC2A','DOCK4','DOK3','DONSON','DPEP1','DPH1','DPPA5','DPYSL2','DQX1','DRD1','DRP2','DSC3','DSEL','DSG2','DSP','DTNA','DUS1L','DUSP15','DXO','DYSF','ECSCR','EDA2R','EDNRA','EDNRB','EEF1AKMT1','EEF2','EFCAB1','EFCAB11','EFCAB2','EFCAB7','EFHC2','EFNB1','EFR3A','EFR3B','EGFR','EHBP1','EHD1','EHD2','EHD3','EHD4','EIF2B2','EIF5','EIF5B','ELFN1','ELMO3','ELN','ELOC','EMP3','ENAH','ENC1','ENG','ENKD1','ENO1','ENO2','ENO3','ENOX1','ENOX2','EPB41','EPB41L1','EPB41L2','EPB41L3','EPB41L4A','EPB41L4B','EPB41L5','EPCAM','EPHA2','EPHA3','EPHB1','EPHB2','EPN1','EPOR','EPS8L2','ERAP1','ERBB2','ERBB3','ERBIN','ERC1','ERCC4','ERMN','ESCO2','ETNK1','ETS2','EVA1A','EVI2B','EXOC1','EXOC3L1','EXOC7','EZR','F11R','F2RL1','F2RL3','FAAP20','FABP2','FABP3','FABP5','FAIM','FAM126A','FAM126B','FAM129A','FAM129B','FAM13A','FAM163B','FAM171A1','FAM193A','FAM205A','FAM210B','FAM222A','FAM45A','FAM47E','FAM47E-STBD1','FAM72A','FAM72B','FAM72C','FAM72D','FANCG','FAS','FASN','FBLIM1','FBLN2','FBLN5','FBLN7','FBP2','FBRSL1','FBXL2','FBXO16','FBXO38','FBXO41','FBXW11','FCAR','FCGBP','FCGR1A','FCGR1B','FCGR2A','FCHSD2','FFAR4','FGD1','FGD5','FGFR2','FGR','FHIT','FHL1','FIBCD1','FIGN','FILIP1','FILIP1L','FKBP2','FLAD1','FLCN','FLNA','FLNB','FLNC','FLOT1','FLOT2','FLRT3','FLT1','FLT4','FLVCR1','FMN2','FMNL2','FMNL3','FNBP1','FOXA2','FOXN3','FP565260.3','FRMD3','FRMD7','FRMD8','FRS2','FSCN1','FTCD','FUT2','FXYD3','FYB1','FYN','FZD1','FZD2','FZD4','FZD6','GAB1','GAB2','GABRA1','GABRA2','GABRA3','GABRA5','GABRG1','GAD1','GADL1','GAGE1','GAGE10','GAGE12C','GAGE12D','GAGE12E','GAGE12F','GAGE12G','GAGE12H','GAGE12J','GAGE13','GAGE2A','GAGE2E','GALNT10','GALR1','GAP43','GAPDH','GAPT','GAPVD1','GAS2','GAS2L1','GAS7','GAS8','GBP4','GCA','GCC1','GCLM','GCSAML-AS1','GFI1B','GFOD1','GFRA3','GHR','GID8','GINM1','GIPC1','GJA1','GJB3','GJB4','GJB6','GJB7','GJC1','GJC2','GLO1','GLP2R','GLRX','GLUL','GMIP','GNAI2','GNAQ','GNAS','GNB1','GNB2','GNB3','GNB4','GNG2','GP9','GPAT3','GPATCH1','GPBP1','GPBP1L1','GPC1','GPC3','GPC4','GPI','GPR1','GPR142','GPR156','GPR157','GPR160','GPR173','GPR20','GPR27','GPR50','GPR63','GPRC5A','GPRIN1','GPRIN2','GPX8','GRAMD1A','GRAMD1B','GRAMD1C','GRAP2','GRB7','GRID2IP','GRIK5','GRK3','GRK5','GRPR','GSDMA','GTF2F1','GTSE1','GUCA1A','GYPA','GYPC','HAPLN3','HARBI1','HAS1','HAUS4','HAUS7','HCAR2','HCAR3','HCFC2','HCK','HCLS1','HCST','HDAC8','HEG1','HELZ','HENMT1','HERC2','HERPUD1','HFE','HIF1A','HIF3A','HINT1','HIP1R','HLA-A','HLA-DPB1','HMGCS1','HMOX1','HOMER3','HOXC5','HOXD4','HPS3','HRASLS5','HRH1','HS6ST3','HSH2D','HSPB1','HSPG2','HTR1B','HTR7','HTRA1','HUNK','HYDIN','HYI','HYLS1','ICAM1','ICAM2','ICOS','ICOSLG','IFFO2','IFIT5','IFNGR1','IFRD1','IFT122','IGFL2','IGSF11','IGSF23','IKZF5','IL10RA','IL12RB1','IL12RB2','IL13','IL13RA2','IL16','IL17RB','IL1R1','IL1RL1','IL21R','IL4R','IL6ST','IL7R','ILDR2','ILK','INPP5A','INSR','INTS12','IP6K2','IPCEF1','IQGAP1','IQGAP2','IREB2','IRF2','IRS1','ISLR','ITGA2B','ITGA3','ITGA4','ITGA7','ITGA8','ITGA9','ITGB1','ITGB2','ITGB3','ITGB4','ITGB5','ITGB6','ITGB7','ITGB8','ITK','ITM2A','ITPKA','ITSN1','ITSN2','IYD','JAG1','JAK2','JAK3','JAM2','JAML','JCAD','JMJD4','JPT2','JUP','KANK1','KANK3','KANSL2','KARS','KATNB1','KCNAB1','KCNB2','KCNH8','KCNJ15','KCNK13','KCNK17','KCNK6','KCNK7','KCNK9','KCNN2','KCNS3','KCNU1','KDF1','KDR','KEL','KIAA0556','KIAA0895','KIAA1324','KIAA1522','KIAA1958','KIF17','KIF19','KIF21A','KIF21B','KIF26A','KIF26B','KISS1R','KIT','KLC1','KLC2','KLF9','KLHDC7B','KLHL41','KLHL6','KLHL7','KLK7','KLRC1','KLRC2','KLRF1','KLRK1','KMT2D','KNDC1','KNSTRN','KNTC1','KPTN','KRT75','KRT77','KRTAP25-1','L1CAM','L3MBTL1','LAIR1','LAMA4','LAMC1','LAMP2','LAMTOR1','LAPTM4A','LASP1','LAT','LAT2','LAX1','LCP1','LCTL','LDB2','LDLRAD3','LENG1','LGALS3','LGALS4','LGALS8','LGR6','LHFPL2','LHX6','LIMA1','LINGO1','LIPJ','LMCD1','LMOD2','LMTK2','LMTK3','LNX1','LNX2','LONRF1','LONRF3','LOXHD1','LPAR1','LPAR3','LPAR4','LPIN1','LPIN2','LRCH2','LRP11','LRP12','LRRC15','LRRC26','LRRC34','LRRC45','LRRC69','LRRC7','LRRC9','LRRFIP2','LRRIQ3','LSMEM2','LSR','LTB4R2','LUZP2','LY6K','LYN','LYPD1','LYST','LZTS1','LZTS2','MACF1','MADD','MAGI1','MAGI3','MAMDC2','MAP1S','MAP2','MAP2K1','MAP2K4','MAP2K7','MAP3K12','MAP3K21','MAP3K6','MAP3K7','MAP4','MAP4K1','MAP4K5','MAP6','MAPK15','MAPK8IP1','MAPT','MARCH10','MARCH7','MARCKS','MARK2','MARVELD2','MAS1L','MAT2B','MBP','MC5R','MCAM','MCEMP1','MCF2L2','MCMBP','MCTS1','MDGA1','ME1','MEF2B','MEIOC','MEPCE','MERTK','MET','METAP1','METAP2','METTL15','METTL25','METTL9','MFAP3L','MFSD1','MFSD2A','MGAT4C','MIB1','MICAL2','MICAL3','MICALL1','MICB','MIEN1','MIGA2','MISP','MKLN1','MKRN3','MLF2','MLKL','MLST8','MMACHC','MMP10','MMP15','MMP24','MMP25','MOAP1','MPHOSPH8','MPHOSPH9','MPI','MPIG6B','MPL','MPP1','MPP5','MPP7','MPZL1','MPZL2','MR1','MRGPRF','MRNIP','MRPL42','MRPL44','MRPL46','MRPS18B','MS4A1','MS4A12','MS4A14','MS4A15','MSN','MSRA','MTCP1','MTFR1L','MTHFR','MTM1','MTPN','MTSS1','MUC1','MUC20','MVD','MYH11','MYH9','MYL4','MYLK','MYO10','MYO1B','MYO1C','MYO1E','MYO1G','MYO3A','MYO5B','MYO9A','MYOF','N4BP2','NAA35','NAB1','NAIF1','NAMPT','NAP1L2','NAPG','NARFL','NCAM1','NCAM2','NCK1','NCKAP1L','NCKAP5','NCKIPSD','NCOA1','NCS1','NDC1','NECTIN2','NECTIN3','NECTIN4','NEDD1','NEDD9','NEO1','NEU1','NEXN','NF2','NFE2L2','NFKBID','NGEF']
MembraneProteins2=['NGFR','NHS','NHSL2','NID2','NINJ2','NKTR','NKX2-8','NLGN1','NLGN3','NLK','NLRX1','NMB','NMBR','NME1','NME2','NMT1','NMT2','NMUR2','NOS1','NOTCH2','NPFFR2','NPL','NPNT','NPR1','NPTN','NPTXR','NR2C1','NRAS','NRCAM','NRP2','NRSN1','NRXN1','NSD1','NSFL1C','NT5E','NTNG1','NTNG2','NTSR1','NUDCD3','NUDT16L1','NUDT9','NUMB','NUP35','NUS1','OAS3','OBSCN','OCA2','OCLN','OCM','OCM2','ODC1','ODR4','OGT','OLFM4','OLIG2','OLR1','OPHN1','OPRK1','OR10P1','OR1N2','OR2C3','OR6B2','OR6B3','OR6C6','OR8S1','OSBPL10','OSBPL6','OSBPL7','OSCAR','OSCP1','OSGEP','OSR2','OTOP2','OTULIN','OXGR1','P3H3','PACSIN2','PACSIN3','PAFAH1B2','PAG1','PAIP1','PAK1','PAK4','PAK6','PALLD','PALM','PALM2','PALM2-AKAP2','PALM3','PALMD','PANX1','PANX2','PAQR8','PARD3','PARD3B','PARD6A','PARD6G','PARP6','PATJ','PAWR','PCBD2','PCDH1','PCDH17','PCDH7','PCDHA1','PCDHA11','PCDHA12','PCDHA4','PCDHA5','PCDHB1','PCDHB11','PCDHB12','PCDHB16','PCDHB4','PCDHB5','PCDHGA1','PCDHGA10','PCDHGA11','PCDHGA12','PCDHGA2','PCDHGA3','PCDHGA4','PCDHGA5','PCDHGA6','PCDHGA7','PCDHGA8','PCDHGA9','PCDHGB1','PCDHGB2','PCDHGB3','PCDHGB4','PCDHGB5','PCDHGB6','PCDHGB7','PCDHGC3','PCDHGC4','PCDHGC5','PCLO','PCMTD1','PDAP1','PDCD7','PDE3A','PDE4A','PDE4D','PDE9A','PDGFC','PDGFRA','PDHX','PDLIM1','PDLIM5','PDPN','PDZD2','PDZD8','PEAR1','PEBP1','PFDN1','PGAP3','PGLYRP2','PGLYRP4','PGRMC2','PHACTR1','PHACTR4','PHF20L1','PHGDH','PHLDB2','PHLDB3','PHOSPHO1','PHPT1','PI3','PI4K2A','PI4KA','PIEZO2','PIGB','PIGN','PIGW','PIGX','PIK3AP1','PIK3CG','PIK3R6','PIP4K2A','PIP4K2C','PIP5K1A','PIRT','PITPNM3','PKD2L2','PKN1','PKN2','PKP1','PKP2','PKP3','PKP4','PLAA','PLAUR','PLCB1','PLCB2','PLCD3','PLCD4','PLEKHA4','PLEKHA6','PLEKHA7','PLEKHG3','PLEKHG4','PLEKHG6','PLEKHH2','PLEKHO2','PLIN4','PLPP1','PLPPR5','PLS3','PLSCR1','PLVAP','PLXNA3','PLXNA4','PNCK','PNISR','PODXL','POLE','POU2F3','PPEF1','PPFIBP1','PPIP5K1','PPL','PPM1A','PPM1F','PPP1CA','PPP1CB','PPP1R12B','PPP1R16A','PPP1R16B','PPP1R18','PPP1R27','PPP1R9A','PPP2R1B','PPP3R1','PPP3R2','PPP4C','PPP6R3','PPTC7','PRAME','PRC1','PRDX6','PRIMA1','PRKCA','PRKCE','PRKCH','PRKCZ','PRKD1','PROM2','PROSER2','PRR15','PRR32','PRR7','PRRC2A','PRRG1','PRRT2','PRRT3','PRRT4','PRTG','PRX','PSCA','PSD','PSD2','PSEN1','PSMC5','PSMC6','PSMD9','PSTPIP2','PTAFR','PTCHD4','PTK2B','PTK6','PTPN12','PTPN21','PTPN22','PTPN4','PTPRB','PTPRG','PTPRK','PTPRM','PTPRR','PTPRS','PTX3','PURG','PVALB','PVR','PXDC1','PXK','PYGL','PYGO2','PYM1','RAB13','RAB23','RAB25','RAB26','RAB27B','RAB3B','RAB3GAP2','RAB43','RAB8A','RABEPK','RAD51C','RAD52','RAI2','RALA','RALGAPA2','RAPGEF2','RAPH1','RAPSN','RASA4','RASA4B','RASAL1','RASAL2','RASAL3','RASD1','RASD2','RASGRP1','RASGRP2','RASGRP4','RASSF2','RASSF3','RASSF4','RBM46','RC3H1','RCCD1','RCSD1','RD3L','RDX','RECK','RELL1','REPS1','RET','RFLNB','RFPL1','RFPL2','RFPL3','RGP1','RGS10','RGS14','RGS19','RGS3','RGS7BP','RGS9BP','RHCE','RHD','RHEX','RHOA','RHOBTB2','RHOJ','RHOXF2','RHOXF2B','RIC8A','RIC8B','RIF1','RILPL1','RIMBP3','RIMBP3B','RIMBP3C','RIMS2','RIOK2','RIOK3','RIPK1','RND1','RNF114','RNF145','RNF146','RNF149','RNF165','RNF166','RNF39','ROBO1','ROBO3','ROBO4','ROM1','RP2','RPE','RPEL1','RPGRIP1L','RPIA','RPL36A','RPL36A-HNRNPH2','RPL36AL','RPSA','RPUSD1','RRAD','RRP12','RRP7A','RSBN1L','RSC1A1','RSPH3','RTL3','RTL4','RTN4R','RTN4RL1','RYR2','S100A13','S100A14','S100A16','S100A2','S100A3','S100A4','S100A6','S100A9','SAMD4A','SAMHD1','SAMSN1','SAPCD1','SASH3','SC5D','SCAMP1','SCAMP4','SCEL','SCHIP1','SCIN','SCN1A','SCN5A','SCN8A','SCN9A','SCNN1D','SCNN1G','SCRIB','SCUBE3','SDC2','SDC4','SDE2','SDHB','SEC16B','SEMA3C','SEMA3D','SEMA4B','SEMA4F','SEMA6C','SEMA6D','SEPT2','SEPT3','SERINC2','SERINC5','SERPINB3','SFSWAP','SGCE','SGK494','SH3BGR','SH3D21','SHANK1','SHANK3','SHARPIN','SHISA6','SHKBP1','SHROOM2','SHTN1','SIGLEC10','SIGLEC14','SIGLEC6','SIGLEC7','SIN3B','SIPA1','SIPA1L1','SIPA1L3','SKAP1','SLAMF6','SLC10A5','SLC12A2','SLC13A3','SLC13A5','SLC14A1','SLC16A1','SLC16A10','SLC16A2','SLC16A3','SLC16A7','SLC16A9','SLC17A5','SLC19A1','SLC19A3','SLC1A5','SLC1A6','SLC22A16','SLC23A3','SLC24A4','SLC26A9','SLC27A1','SLC28A1','SLC2A1','SLC2A11','SLC2A12','SLC2A14','SLC2A2','SLC2A3','SLC30A1','SLC31A2','SLC34A1','SLC34A2','SLC35F5','SLC35G2','SLC36A4','SLC38A5','SLC38A6','SLC38A7','SLC39A10','SLC39A14','SLC39A2','SLC39A6','SLC3A2','SLC40A1','SLC41A3','SLC43A1','SLC43A3','SLC44A2','SLC45A4','SLC46A1','SLC46A3','SLC4A1AP','SLC4A2','SLC4A7','SLC4A8','SLC5A12','SLC5A3','SLC5A6','SLC5A7','SLC6A6','SLC7A14','SLC7A2','SLC7A5','SLC7A6','SLC8A1','SLC9A1','SLC9A2','SLC9A3','SLC9A3R1','SLC9A3R2','SLC9A4','SLC9A5','SLC9B2','SLC9C2','SLCO1A2','SLCO1B3','SLCO4A1','SLCO4C1','SLCO5A1','SLIT1','SLIT3','SLITRK6','SLK','SLX4','SMAGP','SMAP1','SMOC1','SMPD2','SMPX','SNAP23','SNAP25','SNRK','SNTB2','SNTG2','SNX15','SNX32','SNX9','SOCS3','SOCS5','SOD1','SORBS1','SORCS1','SORT1','SOX11','SPAG16','SPANXN4','SPART','SPATA12','SPATC1L','SPERT','SPG11','SPHK1','SPINK13','SPINT1','SPINT2','SPN','SPPL2B','SPPL3','SPRED3','SPRYD3','SPTBN2','SRC','SRCIN1','SRD5A3','SRPK1','SRPRB','SRRM3','SSFA2','SSH1','SSH3','SSTR1','SSTR5','SSX2IP','ST8SIA5','STAB1','STAC','STAC2','STAMBP','STAMBPL1','STARD4','STAT2','STEAP4','STIL','STIP1','STK10','STK16','STK17A','STK17B','STOM','STOML2','STOML3','STX18','STX4','STXBP4','STYK1','SUCLG1','SUCLG2','SUGT1','SUMF2','SUPT6H','SUSD2','SVIL','SWAP70','SYCP2','SYK','SYMPK','SYNC','SYNGR3','SYNPO2L','SYPL2','SYT4','SYTL2','SYVN1','TACC2','TACR2','TACSTD2','TASP1','TAZ','TBC1D10A','TBC1D2','TBC1D24','TBC1D30','TBC1D9B','TBXA2R','TCAF2','TCAF2C','TCHP','TCIM','TDG','TDO2','TDP1','TEC','TEDC2','TEK','TES','TEX14','TEX9','TGFBR1','TGFBR2','TGM1','TGM2','THBS1','THBS2','THEGL','THY1','TIAM1','TICAM1','TIGD7','TIMD4','TJP1','TJP2','TJP3','TLN1','TLR4','TM2D2','TM4SF20','TM4SF4','TM4SF5','TMC5','TMED8','TMEM101','TMEM107','TMEM108','TMEM121B','TMEM139','TMEM141','TMEM164','TMEM176B','TMEM183A','TMEM186','TMEM2','TMEM218','TMEM222','TMEM229A','TMEM245','TMEM251','TMEM254','TMEM258','TMEM266','TMEM268','TMEM50A','TMEM63B','TMPRSS15','TMPRSS2','TMTC2','TNFRSF10D','TNFRSF11A','TNFRSF12A','TNFRSF21','TNFRSF25','TNFRSF8','TNFRSF9','TNFSF13B','TNK1','TNKS1BP1','TNNI2','TOGARAM2','TOLLIP','TOM1','TP53BP2','TPD52L1','TPRN','TRAF7','TRAIP','TRAPPC13','TRAT1','TRDN','TREML1','TRIB1','TRIM16','TRIM16L','TRIM38','TRIM4','TRIM44','TRIM54','TRIM71','TRIML2','TRIP6','TRMT1','TRMT10B','TRMT13','TRMT6','TRO','TRPM1','TRPM4','TRPM7','TRPV2','TRPV3','TSG101','TSGA13','TSHZ3','TSPAN15','TSPAN6','TSPO2','TSR2','TSSK4','TTC17','TTC23','TTC7B','TTLL12','TTLL5','TULP3','TWF2','TXNDC11','TXNIP','TYROBP','TYW3','UAP1','UBA3','UBA52','UBAC1','UBALD1','UBAP1','UBASH3B','UBE2A','UBE2B','UBE2C','UBE2D1','UBE2D2','UBE2D3','UBE2D4','UBE2K','UBE2QL1','UBE2S','UBE3C','UBQLN2','UBR2','UBTD1','UCP3','ULBP1','ULBP2','UNC5A','UNC79','UQCC1','USP14','USP15','USP2','USP20','USP21','USP4','USP53','UTP20','UTRN','VAMP5','VAMP8','VANGL1','VASP','VCAM1','VCPIP1','VIL1','VPS13B','VPS36','VPS50','VPS8','VSIG4','VTCN1','VWA5B1','VWA7','WAS','WASF2','WBP4','WDFY3','WDPCP','WDR1','WDR12','WDR13','WDR53','WDR6','WDR66','WIPF1','WWP1','XIAP','XIRP1','XIRP2','XPC','XPNPEP2','XPO6','XRN1','XYLT1','YES1','YIPF1','YIPF4','YOD1','YTHDC1','ZBED3','ZBTB33','ZBTB8OS','ZDHHC2','ZDHHC20','ZDHHC22','ZDHHC5','ZFP41','ZFP90','ZFYVE28','ZGPAT','ZMYND10','ZNF10','ZNF18','ZNF185','ZNF2','ZNF20','ZNF227','ZNF229','ZNF236','ZNF276','ZNF277','ZNF337','ZNF397','ZNF407','ZNF497','ZNF503','ZNF510','ZNF536','ZNF571','ZNF598','ZNF599','ZNF646','ZNF655','ZNF688','ZNF706','ZNF717','ZNF783','ZNF79','ZNF839','ZSCAN22','ZSCAN29','ZSWIM1','ZSWIM8','ZYX']
MembraneProteins=MembraneProteins1+MembraneProteins2

TFs1=['AC008770.3','AC023509.3','AC092835.1','AC138696.1','ADNP','ADNP2','AEBP1','AEBP2','AHCTF1','AHDC1','AHR','AHRR','AIRE','AKAP8','AKAP8L','AKNA','ALX1','ALX3','ALX4','ANHX','ANKZF1','AR','ARGFX','ARHGAP35','ARID2','ARID3A','ARID3B','ARID3C','ARID5A','ARID5B','ARNT','ARNT2','ARNTL','ARNTL2','ARX','ASCL1','ASCL2','ASCL3','ASCL4','ASCL5','ASH1L','ATF1','ATF2','ATF3','ATF4','ATF5','ATF6','ATF6B','ATF7','ATMIN','ATOH1','ATOH7','ATOH8','BACH1','BACH2','BARHL1','BARHL2','BARX1','BARX2','BATF','BATF2','BATF3','BAZ2A','BAZ2B','BBX','BCL11A','BCL11B','BCL6','BCL6B','BHLHA15','BHLHA9','BHLHE22','BHLHE23','BHLHE40','BHLHE41','BNC1','BNC2','BORCS8-MEF2B','BPTF','BRF2','BSX','C11orf95','CAMTA1','CAMTA2','CARF','CASZ1','CBX2','CC2D1A','CCDC169-SOHLH2','CCDC17','CDC5L','CDX1','CDX2','CDX4','CEBPA','CEBPB','CEBPD','CEBPE','CEBPG','CEBPZ','CENPA','CENPB','CENPBD1','CENPS','CENPT','CENPX','CGGBP1','CHAMP1','CHCHD3','CIC','CLOCK','CPEB1','CPXCR1','CREB1','CREB3','CREB3L1','CREB3L2','CREB3L3','CREB3L4','CREB5','CREBL2','CREBZF','CREM','CRX','CSRNP1','CSRNP2','CSRNP3','CTCF','CTCFL','CUX1','CUX2','CXXC1','CXXC4','CXXC5','DACH1','DACH2','DBP','DBX1','DBX2','DDIT3','DEAF1','DLX1','DLX2','DLX3','DLX4','DLX5','DLX6','DMBX1','DMRT1','DMRT2','DMRT3','DMRTA1','DMRTA2','DMRTB1','DMRTC2','DMTF1','DNMT1','DNTTIP1','DOT1L','DPF1','DPF3','DPRX','DR1','DRAP1','DRGX','DUX1','DUX3','DUX4','DUXA','DZIP1','E2F1','E2F2','E2F3','E2F4','E2F5','E2F6','E2F7','E2F8','E4F1','EBF1','EBF2','EBF3','EBF4','EEA1','EGR1','EGR2','EGR3','EGR4','EHF','ELF1','ELF2','ELF3','ELF4','ELF5','ELK1','ELK3','ELK4','EMX1','EMX2','EN1','EN2','EOMES','EPAS1','ERF','ERG','ESR1','ESR2','ESRRA','ESRRB','ESRRG','ESX1','ETS1','ETS2','ETV1','ETV2','ETV3','ETV3L','ETV4','ETV5','ETV6','ETV7','EVX1','EVX2','FAM170A','FAM200B','FBXL19','FERD3L','FEV','FEZF1','FEZF2','FIGLA','FIZ1','FLI1','FLYWCH1','FOS','FOSB','FOSL1','FOSL2','FOXA1','FOXA2','FOXA3','FOXB1','FOXB2','FOXC1','FOXC2','FOXD1','FOXD2','FOXD3','FOXD4','FOXD4L1','FOXD4L3','FOXD4L4','FOXD4L5','FOXD4L6','FOXE1','FOXE3','FOXF1','FOXF2','FOXG1','FOXH1','FOXI1','FOXI2','FOXI3','FOXJ1','FOXJ2','FOXJ3','FOXK1','FOXK2','FOXL1','FOXL2','FOXM1','FOXN1','FOXN2','FOXN3','FOXN4','FOXO1','FOXO3','FOXO4','FOXO6','FOXP1','FOXP2','FOXP3','FOXP4','FOXQ1','FOXR1','FOXR2','FOXS1','GABPA','GATA1','GATA2','GATA3','GATA4','GATA5','GATA6','GATAD2A','GATAD2B','GBX1','GBX2','GCM1','GCM2','GFI1','GFI1B','GLI1','GLI2','GLI3','GLI4','GLIS1','GLIS2','GLIS3','GLMP','GLYR1','GMEB1','GMEB2','GPBP1','GPBP1L1','GRHL1','GRHL2','GRHL3','GSC','GSC2','GSX1','GSX2','GTF2B','GTF2I','GTF2IRD1','GTF2IRD2','GTF2IRD2B','GTF3A','GZF1','HAND1','HAND2','HBP1','HDX','HELT','HES1','HES2','HES3','HES4','HES5','HES6','HES7','HESX1','HEY1','HEY2','HEYL','HHEX','HIC1','HIC2','HIF1A','HIF3A','HINFP','HIVEP1','HIVEP2','HIVEP3','HKR1','HLF','HLX','HMBOX1','HMG20A','HMG20B','HMGA1','HMGA2','HMGN3','HMX1','HMX2','HMX3','HNF1A','HNF1B','HNF4A','HNF4G','HOMEZ','HOXA1','HOXA10','HOXA11','HOXA13','HOXA2','HOXA3','HOXA4','HOXA5','HOXA6','HOXA7','HOXA9','HOXB1','HOXB13','HOXB2','HOXB3','HOXB4','HOXB5','HOXB6','HOXB7','HOXB8','HOXB9','HOXC10','HOXC11','HOXC12','HOXC13','HOXC4','HOXC5','HOXC6','HOXC8','HOXC9','HOXD1','HOXD10','HOXD11','HOXD12','HOXD13','HOXD3','HOXD4','HOXD8','HOXD9','HSF1','HSF2','HSF4','HSF5','HSFX1','HSFX2','HSFY1','HSFY2','IKZF1','IKZF2','IKZF3','IKZF4','IKZF5','INSM1','INSM2','IRF1','IRF2','IRF3','IRF4','IRF5','IRF6','IRF7','IRF8','IRF9','IRX1','IRX2','IRX3','IRX4','IRX5','IRX6','ISL1','ISL2','ISX','JAZF1','JDP2','JRK','JRKL','JUN','JUNB','JUND','KAT7','KCMF1','KCNIP3','KDM2A','KDM2B','KDM5B','KIN','KLF1','KLF10','KLF11','KLF12','KLF13','KLF14','KLF15','KLF16','KLF17','KLF2','KLF3','KLF4','KLF5','KLF6','KLF7','KLF8','KLF9','KMT2A','KMT2B','L3MBTL1','L3MBTL3','L3MBTL4','LBX1','LBX2','LCOR','LCORL','LEF1','LEUTX','LHX1','LHX2','LHX3','LHX4','LHX5','LHX6','LHX8','LHX9','LIN28A','LIN28B','LIN54','LMX1A','LMX1B','LTF','LYL1','MAF','MAFA','MAFB','MAFF','MAFG','MAFK','MAX','MAZ','MBD1','MBD2','MBD3','MBD4','MBD6','MBNL2','MECOM','MECP2','MEF2A','MEF2B','MEF2C','MEF2D','MEIS1','MEIS2','MEIS3','MEOX1','MEOX2','MESP1','MESP2','MGA','MITF','MIXL1','MKX','MLX','MLXIP','MLXIPL','MNT','MNX1','MSANTD1','MSANTD3','MSANTD4','MSC','MSGN1','MSX1','MSX2','MTERF1','MTERF2','MTERF3','MTERF4','MTF1','MTF2','MXD1','MXD3','MXD4','MXI1','MYB','MYBL1','MYBL2','MYC','MYCL','MYCN','MYF5','MYF6','MYNN','MYOD1','MYOG','MYPOP','MYRF','MYRFL','MYSM1','MYT1','MYT1L','MZF1','NACC2','NAIF1','NANOG','NANOGNB','NANOGP8','NCOA1','NCOA2','NCOA3','NEUROD1','NEUROD2','NEUROD4','NEUROD6','NEUROG1','NEUROG2','NEUROG3','NFAT5','NFATC1','NFATC2','NFATC3','NFATC4','NFE2','NFE2L1','NFE2L2','NFE2L3','NFE4','NFIA','NFIB','NFIC','NFIL3','NFIX','NFKB1','NFKB2','NFX1','NFXL1','NFYA','NFYB','NFYC','NHLH1','NHLH2','NKRF','NKX1-1','NKX1-2','NKX2-1','NKX2-2','NKX2-3','NKX2-4','NKX2-5','NKX2-6','NKX2-8','NKX3-1','NKX3-2','NKX6-1','NKX6-2','NKX6-3','NME2','NOBOX','NOTO','NPAS1','NPAS2','NPAS3','NPAS4','NR0B1','NR1D1','NR1D2','NR1H2','NR1H3','NR1H4','NR1I2','NR1I3','NR2C1','NR2C2','NR2E1','NR2E3','NR2F1','NR2F2','NR2F6','NR3C1','NR3C2','NR4A1','NR4A2','NR4A3','NR5A1','NR5A2','NR6A1','NRF1','NRL','OLIG1','OLIG2','OLIG3','ONECUT1','ONECUT2','ONECUT3','OSR1','OSR2','OTP','OTX1','OTX2','OVOL1','OVOL2','OVOL3','PA2G4','PATZ1','PAX1','PAX2','PAX3','PAX4','PAX5','PAX6','PAX7','PAX8','PAX9','PBX1','PBX2','PBX3','PBX4','PCGF2','PCGF6','PDX1','PEG3','PGR','PHF1','PHF19 ','PHF20','PHF21A','PHOX2A','PHOX2B','PIN1','PITX1','PITX2','PITX3','PKNOX1','PKNOX2','PLAG1','PLAGL1','PLAGL2','PLSCR1','POGK','POU1F1','POU2AF1','POU2F1','POU2F2','POU2F3','POU3F1','POU3F2','POU3F3','POU3F4','POU4F1','POU4F2','POU4F3','POU5F1','POU5F1B','POU5F2','POU6F1','POU6F2','PPARA','PPARD','PPARG','PRDM1','PRDM10','PRDM12','PRDM13','PRDM14','PRDM15','PRDM16','PRDM2','PRDM4','PRDM5','PRDM6','PRDM8','PRDM9','PREB','PRMT3','PROP1','PROX1','PROX2','PRR12','PRRX1','PRRX2','PTF1A','PURA','PURB','PURG','RAG1','RARA','RARB','RARG','RAX','RAX2','RBAK','RBCK1','RBPJ','RBPJL','RBSN','REL','RELA','RELB','REPIN1','REST','REXO4','RFX1','RFX2','RFX3','RFX4','RFX5','RFX6','RFX7','RFX8','RHOXF1','RHOXF2','RHOXF2B','RLF','RORA','RORB','RORC','RREB1','RUNX1','RUNX2','RUNX3','RXRA','RXRB','RXRG','SAFB','SAFB2','SALL1','SALL2','SALL3','SALL4','SATB1','SATB2','SCMH1','SCML4','SCRT1','SCRT2','SCX','SEBOX','SETBP1','SETDB1','SETDB2','SGSM2','SHOX','SHOX2','SIM1','SIM2','SIX1','SIX2','SIX3','SIX4','SIX5','SIX6','SKI','SKIL','SKOR1','SKOR2','SLC2A4RG','SMAD1','SMAD3','SMAD4','SMAD5','SMAD9','SMYD3','SNAI1','SNAI2','SNAI3','SNAPC2','SNAPC4','SNAPC5','SOHLH1','SOHLH2','SON','SOX1','SOX10','SOX11']
TFs2=['SOX12','SOX13','SOX14','SOX15','SOX17','SOX18','SOX2','SOX21','SOX3','SOX30','SOX4','SOX5','SOX6','SOX7','SOX8','SOX9','SP1','SP100','SP110','SP140','SP140L','SP2','SP3','SP4','SP5','SP6','SP7','SP8','SP9','SPDEF','SPEN','SPI1','SPIB','SPIC','SPZ1','SRCAP','SREBF1','SREBF2','SRF','SRY','ST18','STAT1','STAT2','STAT3','STAT4','STAT5A','STAT5B','STAT6','T','TAL1','TAL2','TBP','TBPL1','TBPL2','TBR1','TBX1','TBX10','TBX15','TBX18','TBX19','TBX2','TBX20','TBX21','TBX22','TBX3','TBX4','TBX5','TBX6','TCF12','TCF15','TCF20','TCF21','TCF23','TCF24','TCF3','TCF4','TCF7','TCF7L1','TCF7L2','TCFL5','TEAD1','TEAD2','TEAD3','TEAD4','TEF','TERB1','TERF1','TERF2','TET1','TET2','TET3','TFAP2A','TFAP2B','TFAP2C','TFAP2D','TFAP2E','TFAP4','TFCP2','TFCP2L1','TFDP1','TFDP2','TFDP3','TFE3','TFEB','TFEC','TGIF1','TGIF2','TGIF2LX','TGIF2LY','THAP1','THAP10','THAP11','THAP12','THAP2','THAP3','THAP4','THAP5','THAP6','THAP7','THAP8','THAP9','THRA','THRB','THYN1','TIGD1','TIGD2','TIGD3','TIGD4','TIGD5','TIGD6','TIGD7','TLX1','TLX2','TLX3','TMF1','TOPORS','TP53','TP63','TP73','TPRX1','TRAFD1','TRERF1','TRPS1','TSC22D1','TSHZ1','TSHZ2','TSHZ3','TTF1','TWIST1','TWIST2','UBP1','UNCX','USF1','USF2','USF3','VAX1','VAX2','VDR','VENTX','VEZF1','VSX1','VSX2','WIZ','WT1','XBP1','XPA','YBX1','YBX2','YBX3','YY1','YY2','ZBED1','ZBED2','ZBED3','ZBED4','ZBED5','ZBED6','ZBED9','ZBTB1','ZBTB10','ZBTB11','ZBTB12','ZBTB14','ZBTB16','ZBTB17','ZBTB18','ZBTB2','ZBTB20','ZBTB21','ZBTB22','ZBTB24','ZBTB25','ZBTB26','ZBTB3','ZBTB32','ZBTB33','ZBTB34','ZBTB37','ZBTB38','ZBTB39','ZBTB4','ZBTB40','ZBTB41','ZBTB42','ZBTB43','ZBTB44','ZBTB45','ZBTB46','ZBTB47','ZBTB48','ZBTB49','ZBTB5','ZBTB6','ZBTB7A','ZBTB7B','ZBTB7C','ZBTB8A','ZBTB8B','ZBTB9','ZC3H8','ZEB1','ZEB2','ZFAT','ZFHX2','ZFHX3','ZFHX4','ZFP1','ZFP14','ZFP2','ZFP28','ZFP3','ZFP30','ZFP37','ZFP41','ZFP42','ZFP57','ZFP62','ZFP64','ZFP69','ZFP69B','ZFP82','ZFP90','ZFP91','ZFP92','ZFPM1','ZFPM2','ZFX','ZFY','ZGLP1','ZGPAT','ZHX1','ZHX2','ZHX3','ZIC1','ZIC2','ZIC3','ZIC4','ZIC5','ZIK1','ZIM2','ZIM3','ZKSCAN1','ZKSCAN2','ZKSCAN3','ZKSCAN4','ZKSCAN5','ZKSCAN7','ZKSCAN8','ZMAT1','ZMAT4','ZNF10','ZNF100','ZNF101','ZNF107','ZNF112','ZNF114','ZNF117','ZNF12','ZNF121','ZNF124','ZNF131','ZNF132','ZNF133','ZNF134','ZNF135','ZNF136','ZNF138','ZNF14','ZNF140','ZNF141','ZNF142','ZNF143','ZNF146','ZNF148','ZNF154','ZNF155','ZNF157','ZNF16','ZNF160','ZNF165','ZNF169','ZNF17','ZNF174','ZNF175','ZNF177','ZNF18','ZNF180','ZNF181','ZNF182','ZNF184','ZNF189','ZNF19','ZNF195','ZNF197','ZNF2','ZNF20','ZNF200','ZNF202','ZNF205','ZNF207','ZNF208','ZNF211','ZNF212','ZNF213','ZNF214','ZNF215','ZNF217','ZNF219','ZNF22','ZNF221','ZNF222','ZNF223','ZNF224','ZNF225','ZNF226','ZNF227','ZNF229','ZNF23','ZNF230','ZNF232','ZNF233','ZNF234','ZNF235','ZNF236','ZNF239','ZNF24','ZNF248','ZNF25','ZNF250','ZNF251','ZNF253','ZNF254','ZNF256','ZNF257','ZNF26','ZNF260','ZNF263','ZNF264','ZNF266','ZNF267','ZNF268','ZNF273','ZNF274','ZNF275','ZNF276','ZNF277','ZNF28','ZNF280A','ZNF280B','ZNF280C','ZNF280D','ZNF281','ZNF282','ZNF283','ZNF284','ZNF285','ZNF286A','ZNF286B','ZNF287','ZNF292','ZNF296','ZNF3','ZNF30','ZNF300','ZNF302','ZNF304','ZNF311','ZNF316','ZNF317','ZNF318','ZNF319','ZNF32','ZNF320','ZNF322','ZNF324','ZNF324B','ZNF326','ZNF329','ZNF331','ZNF333','ZNF334','ZNF335','ZNF337','ZNF33A','ZNF33B','ZNF34','ZNF341','ZNF343','ZNF345','ZNF346','ZNF347','ZNF35','ZNF350','ZNF354A','ZNF354B','ZNF354C','ZNF358','ZNF362','ZNF365','ZNF366','ZNF367','ZNF37A','ZNF382','ZNF383','ZNF384','ZNF385A','ZNF385B','ZNF385C','ZNF385D','ZNF391','ZNF394','ZNF395','ZNF396','ZNF397','ZNF398','ZNF404','ZNF407','ZNF408','ZNF41','ZNF410','ZNF414','ZNF415','ZNF416','ZNF417','ZNF418','ZNF419','ZNF420','ZNF423','ZNF425','ZNF426','ZNF428','ZNF429','ZNF43','ZNF430','ZNF431','ZNF432','ZNF433','ZNF436','ZNF438','ZNF439','ZNF44','ZNF440','ZNF441','ZNF442','ZNF443','ZNF444','ZNF445','ZNF446','ZNF449','ZNF45','ZNF451','ZNF454','ZNF460','ZNF461','ZNF462','ZNF467','ZNF468','ZNF469','ZNF470','ZNF471','ZNF473','ZNF474','ZNF479','ZNF48','ZNF480','ZNF483','ZNF484','ZNF485','ZNF486','ZNF487','ZNF488','ZNF490','ZNF491','ZNF492','ZNF493','ZNF496','ZNF497','ZNF500','ZNF501','ZNF502','ZNF503','ZNF506','ZNF507','ZNF510','ZNF511','ZNF512','ZNF512B','ZNF513','ZNF514','ZNF516','ZNF517','ZNF518A','ZNF518B','ZNF519','ZNF521','ZNF524','ZNF525','ZNF526','ZNF527','ZNF528','ZNF529','ZNF530','ZNF532','ZNF534','ZNF536','ZNF540','ZNF541','ZNF543','ZNF544','ZNF546','ZNF547','ZNF548','ZNF549','ZNF550','ZNF551','ZNF552','ZNF554','ZNF555','ZNF556','ZNF557','ZNF558','ZNF559','ZNF560','ZNF561','ZNF562','ZNF563','ZNF564','ZNF565','ZNF566','ZNF567','ZNF568','ZNF569','ZNF57','ZNF570','ZNF571','ZNF572','ZNF573','ZNF574','ZNF575','ZNF576','ZNF577','ZNF578','ZNF579','ZNF580','ZNF581','ZNF582','ZNF583','ZNF584','ZNF585A','ZNF585B','ZNF586','ZNF587','ZNF587B','ZNF589','ZNF592','ZNF594','ZNF595','ZNF596','ZNF597','ZNF598','ZNF599','ZNF600','ZNF605','ZNF606','ZNF607','ZNF608','ZNF609','ZNF610','ZNF611','ZNF613','ZNF614','ZNF615','ZNF616','ZNF618','ZNF619','ZNF620','ZNF621','ZNF623','ZNF624','ZNF625','ZNF626','ZNF627','ZNF628','ZNF629','ZNF630','ZNF639','ZNF641','ZNF644','ZNF645','ZNF646','ZNF648','ZNF649','ZNF652','ZNF653','ZNF654','ZNF655','ZNF658','ZNF66','ZNF660','ZNF662','ZNF664','ZNF665','ZNF667','ZNF668','ZNF669','ZNF670','ZNF671','ZNF672','ZNF674','ZNF675','ZNF676','ZNF677','ZNF678','ZNF679','ZNF680','ZNF681','ZNF682','ZNF683','ZNF684','ZNF687','ZNF688','ZNF689','ZNF69','ZNF691','ZNF692','ZNF695','ZNF696','ZNF697','ZNF699','ZNF7','ZNF70','ZNF700','ZNF701','ZNF703','ZNF704','ZNF705A','ZNF705B','ZNF705D','ZNF705E','ZNF705G','ZNF706','ZNF707','ZNF708','ZNF709','ZNF71','ZNF710','ZNF711','ZNF713','ZNF714','ZNF716','ZNF717','ZNF718','ZNF721','ZNF724','ZNF726','ZNF727','ZNF728','ZNF729','ZNF730','ZNF732','ZNF735','ZNF736','ZNF737','ZNF74','ZNF740','ZNF746','ZNF747','ZNF749','ZNF750','ZNF75A','ZNF75D','ZNF76','ZNF761','ZNF763','ZNF764','ZNF765','ZNF766','ZNF768','ZNF77','ZNF770','ZNF771','ZNF772','ZNF773','ZNF774','ZNF775','ZNF776','ZNF777','ZNF778','ZNF780A','ZNF780B','ZNF781','ZNF782','ZNF783','ZNF784','ZNF785','ZNF786','ZNF787','ZNF788','ZNF789','ZNF79','ZNF790','ZNF791','ZNF792','ZNF793','ZNF799','ZNF8','ZNF80','ZNF800','ZNF804A','ZNF804B','ZNF805','ZNF808','ZNF81','ZNF813','ZNF814','ZNF816','ZNF821','ZNF823','ZNF827','ZNF829','ZNF83','ZNF830','ZNF831','ZNF835','ZNF836','ZNF837','ZNF84','ZNF841','ZNF843','ZNF844','ZNF845','ZNF846','ZNF85','ZNF850','ZNF852','ZNF853','ZNF860','ZNF865','ZNF878','ZNF879','ZNF880','ZNF883','ZNF888','ZNF891','ZNF90','ZNF91','ZNF92','ZNF93','ZNF98','ZNF99','ZSCAN1','ZSCAN10','ZSCAN12','ZSCAN16','ZSCAN18','ZSCAN2','ZSCAN20','ZSCAN21','ZSCAN22','ZSCAN23','ZSCAN25','ZSCAN26','ZSCAN29','ZSCAN30','ZSCAN31','ZSCAN32','ZSCAN4','ZSCAN5A','ZSCAN5B','ZSCAN5C','ZSCAN9','ZUFSP','ZXDA','ZXDB','ZXDC','ZZZ3']
TFs=TFs1+TFs2

#TFs other, inclusive of the above list, not necessarily DNA binding proteins but affect transcription
TFs1Other=['AATF','ABL1','ABRA','AC008770.3','AC020915.1','AC023509.3','AC092835.1','AC105001.2','AC138696.1','ACOD1','ACTR3B','ACTR3C','ADAM8','ADAMTS17','ADAMTS19','ADAR','ADGRA1','ADGRG3','ADNP','ADNP2','AEBP1','AEBP2','AES','AFAP1 ','AFF1','AFF3','AFF4','AGER','AGT','AHCTF1','AHDC1','AHR','AHRR','AIM2','AIRE','AKAP8','AKAP8L','AKNA','AKT1','ALK','ALKBH2','ALX1','ALX3','ALX4','AMH','ANAPC2','ANHX','ANKRD30A','ANKZF1','ANXA3','ANXA4','AP5Z1','APBB3','APC2','APEX1','APOA4','APOBEC3A','APOBEC3B','APOBEC3G','APTX','AR','ARGFX','ARHGAP35','ARHGEF2','ARHGEF5','ARID1A','ARID1B','ARID2','ARID3A','ARID3B','ARID3C','ARID4A','ARID4B','ARID5A','ARID5B','ARNT','ARNT2','ARNTL','ARNTL2','ARRB1','ARRB2','ARX','ASCC1','ASCL1','ASCL2','ASCL3','ASCL4','ASCL5','ASH1L','ATF1','ATF2','ATF3','ATF4','ATF5','ATF6','ATF6B','ATF7','ATF7IP','ATM','ATMIN','ATOH1','ATOH7','ATOH8','ATR','ATRX','ATXN7','BACH1','BACH2','BAG1','BANF1','BANP','BARHL1','BARHL2','BARX1','BARX2','BATF','BATF2','BATF3','BAZ2A','BAZ2B','BBX','BCL10','BCL11A','BCL11B','BCL3','BCL6','BCL6B','BCLAF1','BCOR','BDP1','BEND4','BEX1','BHLHA15','BHLHA9','BHLHE22','BHLHE23','BHLHE40','BHLHE41','BHMG1','BIN1','BLM','BLZF1','BMI1','BMP2','BMP4','BMP7','BMPR1A','BNC1','BNC2','BOD1L1','BOLA1','BOLA2','BOLA2B','BOLA3','BORCS8-MEF2B','BPNT1','BPTF','BRCA1','BRD4','BRD7','BRD8','BRD9','BRF1','BRF2','BRMS1','BRPF1','BSX','BTAF1','BTBD8','BTG2','BTK','BTRC','BUD31','C11orf95','C14orf39','C17orf49','C1D','C20orf194','C3orf33','CACTIN','CAMK1D','CAMK2A','CAMTA1','CAMTA2','CAPN15','CAPN3','CARD11','CARD14','CARF','CARHSP1','CASP8AP2','CASZ1','CAT','CBFA2T2','CBFA2T3','CBFB','CBL','CBLL1','CBX2','CC2D1A','CC2D1B','CCDC124','CCDC169-SOHLH2','CCDC17','CCDC3','CCDC71','CCDC83','CCNT1','CCNT2','CD36','CD40','CD40LG','CDC5L','CDC6','CDK5RAP3','CDK7','CDK8','CDK9','CDKN1A','CDKN2A','CDX1','CDX2','CDX4','CEBPA','CEBPB','CEBPD','CEBPE','CEBPG','CEBPZ','CELF3','CENPA','CENPB','CENPBD1','CENPS','CENPT','CENPX','CERS2','CERS3','CERS4','CERS5','CERS6','CFLAR','CGGBP1','CHAMP1','CHCHD3','CHD1','CHD2','CHD3','CHD4','CHD6','CHD7','CHD9','CHP1','CHRAC1','CHUK','CHURC1','CIB1','CIC','CIITA','CIR1','CITED1','CITED2','CITED4','CIZ1','CLOCK','CLU','CMKLR1','CNBP','CNOT3','CNOT4','CNOT7','CNOT8','COMMD1','COMMD6','COMMD7','COPS2','COPS3','COPS4','CPEB1','CPSF4','CPSF4L','CPXCR1','CRAMP1','CREB1','CREB3','CREB3L1','CREB3L2','CREB3L3','CREB3L4','CREB5','CREBBP','CREBL2','CREBRF','CREBZF','CREG1','CREM','CRTC1','CRTC2','CRTC3','CRX','CSDC2','CSDE1','CSRNP1','CSRNP2','CSRNP3','CTBP1','CTBP2','CTCF','CTCFL','CTH','CTNNB1','CTNNBIP1','CUL1','CUL2','CUL3','CUL4A','CUL4B','CUL5','CUX1','CUX2','CXXC1','CXXC4','CXXC5','CYLD','CYP1B1','CYTL1','DAB2IP','DACH1','DACH2','DAP','DAXX','DBP','DBX1','DBX2','DCP1A','DDB1','DDB2','DDIT3','DDN','DDR2','DDRGK1','DDX3X','DDX5','DDX58','DEAF1','DEK','DEPDC1','DEPDC1B','DEPDC4','DEPDC5','DEPDC7','DEPTOR','DHX34','DHX36','DHX57','DISP3','DLX1','DLX2','DLX3','DLX4','DLX5','DLX6','DMAP1','DMBX1','DMC1','DMRT1','DMRT2','DMRT3','DMRTA1','DMRTA2','DMRTB1','DMRTC1','DMRTC2','DMTF1','DNA2','DNAJA3','DNAJC1','DNAJC2','DNAJC21','DNMT1','DNMT3A','DNMT3B','DNMT3L','DNTT','DNTTIP1','DOT1L','DPF1','DPF2','DPF3','DPRX','DR1','DRAP1','DRGX','DSP','DTX1','DUS3L','DUSP12','DUX1','DUX3','DUX4','DUXA','DVL1','DVL2','DVL3','DZIP1','DZIP1L','E2F1','E2F2','E2F3','E2F4','E2F5','E2F6','E2F7','E2F8','E4F1','EAF2','EBF1','EBF2','EBF3','EBF4','EDA','EDA2R','EDF1','EDN1','EEA1','EED','EGLN1','EGR1','EGR2','EGR3','EGR4','EHF','EID1','EID2','EIF2AK2','EIF2AK4','EIF3C','EIF3CL','EIF3K','ELF1','ELF2','ELF3','ELF4','ELF5','ELK1','ELK3','ELK4','ELMSAN1','EME1','EMX1','EMX2','EN1','EN2','ENO1','EOMES','EP300','EP400','EPAS1','EPHA5','EPM2AIP1','ERBIN','ERC1','ERCC3','ERCC4','ERF','ERG','ESR1','ESR2','ESRRA','ESRRB','ESRRG','ESX1','ETS1','ETS2','ETV1','ETV2','ETV3','ETV3L','ETV4','ETV5','ETV6','ETV7','EVX1','EVX2','EWSR1','EXO1','EXOC2','EXOG','EZH1','EZH2','FAAP24','FAM153C','FAM170A','FAM171B','FAM200A','FAM200B','FAN1','FANCA','FANCD2','FANK1','FARSA','FARSB','FBN1','FBXL19','FBXO41','FEN1','FER','FERD3L','FEV','FEZF1','FEZF2','FGD1','FHL2','FIGLA','FIZ1','FLI1','FLNA','FLYWCH1','FMNL2','FOS','FOSB','FOSL1','FOSL2','FOXA1','FOXA2','FOXA3','FOXB1','FOXB2','FOXC1','FOXC2','FOXD1','FOXD2','FOXD3','FOXD4','FOXD4L1','FOXD4L2','FOXD4L3','FOXD4L4','FOXD4L5','FOXD4L6','FOXE1','FOXE3','FOXF1','FOXF2','FOXG1','FOXH1','FOXI1','FOXI2','FOXI3','FOXJ1','FOXJ2','FOXJ3','FOXK1','FOXK2','FOXL1','FOXL2','FOXM1','FOXN1','FOXN2','FOXN3','FOXN4','FOXO1','FOXO3','FOXO4','FOXO6','FOXP1','FOXP2','FOXP3','FOXP4','FOXQ1','FOXR1','FOXR2','FOXS1','FUBP1','FUBP3','FUS','FZD1','FZD2','FZD4','FZD6','GABPA','GABPB1','GABPB2','GADD45A','GAS6','GAS7','GATA1','GATA2','GATA3','GATA4','GATA5','GATA6','GATAD1','GATAD2A','GATAD2B','GBX1','GBX2','GCFC2','GCM1','GCM2','GEN1','GFI1','GFI1B','GLA','GLI1','GLI2','GLI3','GLI4','GLIS1','GLIS2','GLIS3','GLMP','GLYR1','GMEB1','GMEB2','GOLGB1','GON4L','GPATCH8','GPBP1','GPBP1L1','GPER1','GPR155','GREM1','GRHL1','GRHL2','GRHL3','GRM6','GSC','GSC2','GSX1','GSX2','GTF2A1','GTF2A1L','GTF2A2','GTF2B','GTF2E2','GTF2F1','GTF2F2','GTF2H1','GTF2H2','GTF2H2C','GTF2H3','GTF2H4','GTF2I','GTF2IRD1','GTF2IRD2','GTF2IRD2B','GTF3A','GTF3C1','GTF3C2','GULP1','GZF1','H1F0','H1FOO','H1FX','HAND1','HAND2','HAVCR2','HBP1','HCFC1','HCK','HDAC1','HDAC10','HDAC2','HDAC3','HDAC4','HDAC5','HDAC6','HDAC7','HDAC8','HDAC9','HDX','HELT','HELZ','HELZ2','HES1','HES2','HES3','HES4','HES5','HES6','HES7','HESX1','HEY1','HEY2','HEYL','HHEX','HIC1','HIC2','HIF1A','HIF3A','HINFP','HIPK2','HIRA','HIST1H1A','HIST1H1B','HIST1H1C','HIST1H1D','HIST1H1E','HIST1H1T','HIVEP1','HIVEP2','HIVEP3','HKR1','HLA-DQB1','HLA-DQB2','HLA-DRB1','HLA-DRB3','HLA-DRB5','HLF','HLTF','HLX','HMBOX1','HMG20A','HMG20B','HMGA1','HMGA2','HMGB1','HMGB2','HMGB3','HMGB4','HMGN1','HMGN3','HMGXB3','HMGXB4','HMOX1','HMX1','HMX2','HMX3','HNF1A','HNF1B','HNF4A','HNF4G','HNRNPA1','HNRNPA3','HNRNPAB','HNRNPCL2','HNRNPD','HNRNPDL','HNRNPK','HNRNPL','HNRNPU','HOMEZ','HOOK2','HOPX','HOXA1','HOXA10','HOXA11','HOXA13','HOXA2','HOXA3','HOXA4','HOXA5','HOXA6','HOXA7','HOXA9','HOXB1','HOXB13','HOXB2','HOXB3','HOXB4','HOXB5','HOXB6','HOXB7','HOXB8','HOXB9','HOXC10','HOXC11','HOXC12','HOXC13','HOXC4','HOXC5','HOXC6','HOXC8','HOXC9','HOXD1','HOXD10','HOXD11','HOXD12','HOXD13','HOXD3','HOXD4','HOXD8','HOXD9','HP1BP3','HR','HSBP1','HSF1','HSF2','HSF4','HSF5','HSFX1','HSFX2','HSFY1','HSFY2','HSPA1A','HSPA1B','ICAM1','ID1','ID2','ID3','ID4','IFI16','IFNB1','IGHM','IKBKB','IKBKG','IKZF1','IKZF2','IKZF3','IKZF4','IKZF5','IL10','IL1B','IL1RAP','IL4','IL5','IL6','INF2','ING4','INS','INSM1','INSM2','IQGAP1','IRAK1','IRAK2','IRAK3','IRF1','IRF2','IRF2BP1','IRF3','IRF4','IRF5','IRF6','IRF7','IRF8','IRF9','IRX1','IRX2','IRX3','IRX4','IRX5','IRX6','ISL1','ISL2','ISX','ITCH','ITGB2','ITGB3BP','IZUMO2','JADE1','JAK2','JARID2','JAZF1','JDP2','JMY','JRK','JRKL','JUN','JUNB','JUND','JUP','KANSL2','KAT2A','KAT2B','KAT5','KAT6A','KAT6B','KAT7','KAT8','KCMF1','KCNIP3','KDM1A','KDM2A','KDM2B','KDM3A','KDM5A','KDM5B','KDM5C','KDM5D','KEAP1','KIAA1549','KIAA1683','KIAA1841','KIAA1958','KIF15','KIN','KIT','KLF1','KLF10','KLF11','KLF12','KLF13','KLF14','KLF15','KLF16','KLF17','KLF2','KLF3','KLF4','KLF5','KLF6','KLF7','KLF8','KLF9','KMT2A','KMT2B','KMT2C','KMT2D','KNTC1','KRAS','KRBOX1','KRT13','KRTAP5-1','KRTAP5-9','KRTAP9-9','L3MBTL1','L3MBTL3','L3MBTL4','LARP1','LARP1B','LARP4','LARP4B','LARP6','LARP7','LBX1','LBX2','LCOR','LCORL','LEF1','LEUTX','LGALS9','LGR4','LHX1','LHX2','LHX3','LHX4','LHX5','LHX6','LHX8','LHX9','LIG1','LIG3','LIN28A','LIN28B','LIN54','LMO2','LMO4','LMX1A','LMX1B','LPP','LRP5','LRP6','LRP8','LRRFIP1','LRRFIP2','LSR','LTF','LTN1','LYAR','LYL1','LZTFL1','LZTR1','LZTS1','MACC1','MACF1','MAD2L2','MAEL','MAF','MAFA','MAFB','MAFF','MAFG','MAFK','MALT1','MAML1','MAML3','MAP2K5','MAP3K10','MAP3K13','MAP3K7','MAPK1','MAPK10','MAPK11','MAPK14','MAPK3','MAPK8','MAPK9','MATR3','MAVS','MAX','MAZ','MB21D1','MBD1','MBD2','MBD3','MBD4','MBD6','MBNL1','MBNL2','MBNL3','MBTPS2','MDFI','MDM2','MDM4','MECOM','MECP2','MED1','MED12','MED13','MED13L','MED14','MED15','MED16','MED17','MED21','MED23','MED24','MED6','MEF2A','MEF2B','MEF2C','MEF2D','MEIS1','MEIS2','MEIS3','MEN1','MEOX1','MEOX2','MESP1','MESP2','MET','MGA','MGMT','MID2','MIER1','MIER2','MIER3','MINK1','MIS18BP1','MITF','MIXL1','MKL1','MKL2','MKRN1','MKRN2','MKRN3','MKX','MLLT1','MLLT10','MLLT11','MLLT3','MLLT6','MLX','MLXIP','MLXIPL','MN1','MNAT1','MNT','MNX1','MORC3','MPG','MRPL28','MRRF','MSANTD1','MSANTD3','MSANTD4','MSC','MSGN1','MSH2','MSH3','MSH6','MSL3','MSRB2','MST1R','MSX1','MSX2','MTA1','MTA2','MTA3','MTDH','MTERF1','MTERF2','MTERF3','MTERF4','MTF1','MTF2','MTPN','MUS81','MXD1','MXD3','MXD4','MXI1','MYB','MYBBP1A','MYBL1','MYBL2','MYC','MYCBP','MYCL','MYCN','MYD88','MYEF2','MYF5','MYF6','MYNN','MYOCD','MYOD1','MYOG','MYPOP','MYRF','MYRFL','MYSM1','MYT1','MYT1L','MZF1','NAB2','NABP2','NACC2','NAIF1','NANOG','NANOGNB','NANOGP8','NCOA1','NCOA2','NCOA3','NCOA4','NCOA5','NCOA6','NCOA7','NCOR1','NCOR2','NDC80','NDN','NDP','NEB','NEIL1','NEUROD1','NEUROD2','NEUROD4','NEUROD6','NEUROG1','NEUROG2','NEUROG3','NFAM1','NFAT5','NFATC1','NFATC2','NFATC3','NFATC4','NFE2','NFE2L1','NFE2L2','NFE2L3','NFE4','NFIA','NFIB','NFIC','NFIL3','NFIX','NFKB1','NFKB2','NFKBIA','NFKBIB','NFKBID','NFKBIL1','NFKBIZ','NFRKB','NFX1','NFXL1','NFYA','NFYB','NFYC','NHLH1','NHLH2','NKRF','NKX1-1','NKX1-2','NKX2-1','NKX2-2','NKX2-3','NKX2-4','NKX2-5','NKX2-6','NKX2-8','NKX3-1','NKX3-2','NKX6-1','NKX6-2','NKX6-3','NLRC3','NLRC4','NLRC5','NLRP12','NLRP2B','NLRP3','NME2','NOBOX','NOC3L','NOC4L','NOCT','NOD1','NOD2','NODAL','NONO','NOP2','NOTCH1','NOTCH2','NOTCH3','NOTO','NPAS1','NPAS2','NPAS3','NPAS4','NPM1','NR0B1','NR0B2','NR1D1','NR1D2','NR1H2','NR1H3','NR1H4','NR1I2','NR1I3','NR2C1','NR2C2','NR2E1','NR2E3','NR2F1','NR2F2','NR2F6','NR3C1','NR3C2','NR4A1','NR4A2','NR4A3','NR5A1','NR5A2','NR6A1','NRBF2','NRF1','NRG1','NRIP1','NRIP3','NRK','NRL','NSD1','NSD2','NTRK1','NUCKS1','NUFIP1','NUP98','NUPL2']
TFs2Other=['OGG1','OLIG1','OLIG2','OLIG3','ONECUT1','ONECUT2','ONECUT3','OPRD1','OSR1','OSR2','OTOP3','OTP','OTULIN','OTX1','OTX2','OVOL1','OVOL2','OVOL3','PA2G4','PABPN1','PARK7','PARP1','PARP10','PARP12','PATZ1','PAWR','PAX1','PAX2','PAX3','PAX4','PAX5','PAX6','PAX7','PAX8','PAX9','PAXBP1','PAXIP1','PBRM1','PBX1','PBX2','PBX3','PBX4','PCBD1','PCBP1','PCBP2','PCBP3','PCGF2','PCGF6','PCNA','PCSK6','PDCD7','PDS5B','PDX1','PEG3','PELI1','PER1','PER2','PER3','PEX14','PFDN1','PGBD1','PGR','PGS1','PHB','PHB2','PHF1','PHF19 ','PHF20','PHF21A','PHF5A','PHOX2A','PHOX2B','PHTF1','PHTF2','PIAS1','PIAS2','PIAS3','PIAS4','PIDD1','PIKFYVE','PIM1','PIN1','PINK1','PITX1','PITX2','PITX3','PKHD1','PKHD1L1','PKNOX1','PKNOX2','PLA2G10','PLA2G1B','PLAG1','PLAGL1','PLAGL2','PLEK','PLEK2','PLEKHA4','PLPP3','PLSCR1','PLXNA1','PLXNA2','PLXNA3','PLXNA4','PLXNB1','PLXNB2','PLXNB3','PLXNC1','PLXND1','PML','PMS1','PNKP','POGK','POGZ','POLA1','POLB','POLE3','POLE4','POLG','POLH','POLI','POLK','POLL','POLM','POLN','POLQ','POLR1B','POLR2A','POLR2B','POLR2E','POLR2J','POLR2K','POLR2L','POLR3A','POLRMT','POT1','POU1F1','POU2AF1','POU2F1','POU2F2','POU2F3','POU3F1','POU3F2','POU3F3','POU3F4','POU4F1','POU4F2','POU4F3','POU5F1','POU5F1B','POU5F2','POU6F1','POU6F2','PPARA','PPARD','PPARG','PPARGC1A','PPARGC1B','PPP1R10','PPP1R13L','PPP1R21','PPP2R3B','PPP2R5B','PPP3CA','PPRC1','PQBP1','PRAM1','PRAP1','PRB3','PRDM1','PRDM10','PRDM11','PRDM12','PRDM13','PRDM14','PRDM15','PRDM16','PRDM2','PRDM4','PRDM5','PRDM6','PRDM7','PRDM8','PRDM9','PRDX3','PREB','PREX1','PREX2','PRH2','PRIM2','PRIMPOL','PRKCB','PRKCH','PRKCI','PRKCQ','PRKCZ','PRKD1','PRKD2','PRKN','PRMT2','PRMT3','PRNP','PROP1','PROX1','PROX2','PRR12','PRR3','PRRX1','PRRX2','PSMA6','PSMC5','PSMD10','PSMD11','PSMD12','PSPC1','PTCH1','PTEN','PTF1A','PTGIS','PTH','PTTG1','PUM3','PURA','PURB','PURG','PYCARD','PYDC1','PYDC2','QRICH1','RAB11FIP4','RAB7B','RAD21','RAD51','RAG1','RAI1','RAPGEF3','RAPGEF4','RAPGEF5','RARA','RARB','RARG','RAX','RAX2','RB1','RBAK','RBBP7','RBCK1','RBL1','RBL2','RBM10','RBM20','RBM22','RBM26','RBM27','RBM5','RBM6','RBPJ','RBPJL','RBSN','RC3H1','RC3H2','RCAN1','RCOR1','RCOR2','RCOR3','RECQL','REL','RELA','RELB','RELL2','RELN','REPIN1','RERE','REST','REV1','REXO4','RFC1','RFX1','RFX2','RFX3','RFX4','RFX5','RFX6','RFX7','RFX8','RFXANK','RFXAP','RGCC','RGS11','RGS6','RGS7','RGS9','RHEBL1','RHOXF1','RHOXF2','RHOXF2B','RING1','RIOK2','RIPK1','RIPK2','RIPK3','RIPK4','RLF','RLIM','RNASE2','RNASEH1','RNF113A','RNF113B','RNF114','RNF125','RNF138','RNF14','RNF166','RNF2','RNF219','RNF25','RNF31','RNF4','RNF41','RORA','RORB','RORC','RPA2','RPA4','RPL36A','RPL7','RPL7A','RPS27A','RPS6KA4','RPS6KA5','RREB1','RRN3','RUNX1','RUNX1T1','RUNX2','RUNX3','RWDD3','RXRA','RXRB','RXRG','RYBP','S100A12','S100A8','S100A9','SAFB','SAFB2','SALL1','SALL2','SALL3','SALL4','SART1','SATB1','SATB2','SAV1','SCAND1','SCAND2P','SCAPER','SCMH1','SCML1','SCML2','SCML4','SCOC','SCRT1','SCRT2','SCX','SEBOX','SEMA4A','SERTAD1','SERTAD2','SETBP1','SETD6','SETDB1','SETDB2','SF3A2','SF3A3','SFPQ','SFRP4','SFRP5','SFTPB','SGK1','SGK3','SGSM2','SHH','SHOX','SHOX2','SHPRH','SIGIRR','SIK1','SIM1','SIM2','SIN3A','SIN3B','SIRT1','SIRT2','SIRT3','SIRT6','SIVA1','SIX1','SIX2','SIX3','SIX4','SIX5','SIX6','SKI','SKIDA1','SKIL','SKOR1','SKOR2','SLC22A4','SLC26A10','SLC26A3','SLC2A4RG','SLC30A9','SLC39A10','SLC4A10','SMAD1','SMAD2','SMAD3','SMAD4','SMAD5','SMAD6','SMAD7','SMAD9','SMARCA1','SMARCA2','SMARCA4','SMARCA5','SMARCAL1','SMARCB1','SMARCC1','SMARCC2','SMARCD1','SMARCD2','SMARCD3','SMARCE1','SMC3','SMO','SMUG1','SMYD1','SMYD3','SNAI1','SNAI2','SNAI3','SNAPC2','SNAPC4','SNAPC5','SNRPC','SNURF','SNW1','SOGA3','SOHLH1','SOHLH2','SON','SORBS2','SOX1','SOX10','SOX11','SOX12','SOX13','SOX14','SOX15','SOX17','SOX18','SOX2','SOX21','SOX3','SOX30','SOX4','SOX5','SOX6','SOX7','SOX8','SOX9','SP1','SP100','SP110','SP140','SP140L','SP2','SP3','SP4','SP5','SP6','SP7','SP8','SP9','SPDEF','SPEN','SPHK1','SPI1','SPIB','SPIC','SPZ1','SRCAP','SREBF1','SREBF2','SRF','SRY','SSB','SSBP2','SSBP3','SSBP4','SSH1','SSH2','SSH3','SSRP1','SSX1','ST18','STAG1','STAT1','STAT2','STAT3','STAT4','STAT5A','STAT5B','STAT6','STK16','STK3','STK36','STRN','STRN3','SUB1','SUFU','SUMO1','SUPT16H','SUPT4H1','SUPT5H','SUPT6H','SUV39H1','SUV39H2','SUZ12','SYK','T','TAB1','TAB2','TAB3','TADA2A','TADA2B','TADA3','TAF1','TAF10','TAF11','TAF12','TAF13','TAF1A','TAF1B','TAF1C','TAF1L','TAF2','TAF3','TAF4','TAF4B','TAF5','TAF5L','TAF6','TAF6L','TAF7','TAF7L','TAF8','TAF9','TAF9B','TAL1','TAL2','TARDBP','TAX1BP1','TBL1Y','TBP','TBPL1','TBPL2','TBR1','TBX1','TBX10','TBX15','TBX18','TBX19','TBX2','TBX20','TBX21','TBX22','TBX3','TBX4','TBX5','TBX6','TCEA2','TCEAL1','TCEAL7','TCEAL8','TCF12','TCF15','TCF19','TCF20','TCF21','TCF23','TCF24','TCF25','TCF3','TCF4','TCF7','TCF7L1','TCF7L2','TCFL5','TDG','TDP1','TDP2','TEAD1','TEAD2','TEAD3','TEAD4','TEF','TERB1','TERF1','TERF2','TERF2IP','TESMIN','TET1','TET2','TET3','TFAM','TFAP2A','TFAP2B','TFAP2C','TFAP2D','TFAP2E','TFAP4','TFCP2','TFCP2L1','TFDP1','TFDP2','TFDP3','TFE3','TFEB','TFEC','TGFB1','TGFB1I1','TGIF1','TGIF2','TGIF2LX','TGIF2LY','TGS1','THAP1','THAP10','THAP11','THAP12','THAP2','THAP3','THAP4','THAP5','THAP6','THAP7','THAP8','THAP9','THRA','THRAP3','THRB','THYN1','TIA1','TICAM1','TIGD1','TIGD2','TIGD3','TIGD4','TIGD5','TIGD6','TIGD7','TIPARP','TIRAP','TLE1','TLE2','TLE3','TLE4','TLE6','TLR2','TLR3','TLR4','TLR9','TLX1','TLX2','TLX3','TMEM235','TMF1','TNF','TNFAIP3','TNFRSF11A','TNFRSF4','TNFSF11','TNFSF18','TNFSF4','TOE1','TONSL','TOP1','TOP2A','TOP2B','TOPORS','TOX','TOX2','TOX3','TOX4','TP53','TP53BP1','TP63','TP73','TPRX1','TRADD','TRAF1','TRAF2','TRAF3','TRAF4','TRAF5','TRAF6','TRAFD1','TRAK2','TRAPPC9','TRERF1','TREX1','TRIB1','TRIB3','TRIM13','TRIM14','TRIM15','TRIM21','TRIM22','TRIM23','TRIM24','TRIM25','TRIM26','TRIM27','TRIM28','TRIM29','TRIM3','TRIM31','TRIM32','TRIM33','TRIM34','TRIM37','TRIM38','TRIM40','TRIM5','TRIM52','TRIM62','TRIM8','TRIP11','TRIT1','TRMT1','TRMT1L','TRPS1','TRRAP','TSC2','TSC22D1','TSC22D2','TSC22D3','TSC22D4','TSHZ1','TSHZ2','TSHZ3','TSN','TSSK4','TTF1','TUB','TULP4','TUT1','TWIST1','TWIST2','TXK','U2AF1','U2AF1L4','UBA52','UBB','UBC','UBE2K','UBE2N','UBE2V1','UBN1','UBP1','UBR4','UBTF','UBTFL1','UBTFL6','UFL1','UHRF1','UHRF2','UNCX','UNG','UNK','UNKL','USF1','USF2','USF3','USP39','USP7','UTF1','VAV1','VAX1','VAX2','VDR','VEGFA','VENTX','VEZF1','VGLL1','VGLL2','VGLL3','VHLL','VPS72','VSX1','VSX2','WDHD1','WFS1','WIZ','WNT1','WNT10B','WNT2','WNT3A','WNT5A','WNT8B','WRN','WT1','WWOX','WWP2','WWTR1','XBP1','XCL1','XPA','YBX1','YBX2','YBX3','YEATS2','YEATS4','YOD1','YY1','YY2','ZBED1','ZBED2','ZBED3','ZBED4','ZBED5','ZBED6','ZBED6CL','ZBED8','ZBED9','ZBP1','ZBTB1','ZBTB10','ZBTB11','ZBTB12','ZBTB14','ZBTB16','ZBTB17','ZBTB18','ZBTB2','ZBTB20','ZBTB21','ZBTB22','ZBTB24','ZBTB25','ZBTB26','ZBTB3','ZBTB32','ZBTB33','ZBTB34','ZBTB37','ZBTB38','ZBTB39','ZBTB4','ZBTB40','ZBTB41','ZBTB42','ZBTB43','ZBTB44','ZBTB45','ZBTB46','ZBTB47','ZBTB48','ZBTB49','ZBTB5','ZBTB6','ZBTB7A','ZBTB7B','ZBTB7C','ZBTB8A','ZBTB8B','ZBTB9','ZC3H10','ZC3H11A','ZC3H12A','ZC3H13','ZC3H14','ZC3H15','ZC3H18','ZC3H3','ZC3H4','ZC3H6','ZC3H7A','ZC3H7B','ZC3H8','ZC3HAV1','ZCCHC11','ZCCHC6','ZDHHC11','ZDHHC19','ZEB1','ZEB2','ZFAND5','ZFAT','ZFHX2','ZFHX3','ZFHX4','ZFP1','ZFP14','ZFP2','ZFP28','ZFP3','ZFP30','ZFP36','ZFP36L1','ZFP36L2','ZFP37','ZFP41','ZFP42','ZFP57','ZFP62','ZFP64','ZFP69','ZFP69B','ZFP82','ZFP90','ZFP91','ZFP92','ZFPL1','ZFPM1','ZFPM2','ZFR','ZFR2','ZFX','ZFY','ZFYVE26','ZGLP1','ZGPAT','ZHX1','ZHX2','ZHX3','ZIC1','ZIC2','ZIC3','ZIC4','ZIC5','ZIK1','ZIM2','ZIM3','ZKSCAN1','ZKSCAN2','ZKSCAN3','ZKSCAN4','ZKSCAN5','ZKSCAN7','ZKSCAN8','ZMAT1','ZMAT2','ZMAT3','ZMAT4','ZMAT5','ZMYM1','ZMYM2','ZMYM3','ZMYM4','ZMYM5','ZMYM6','ZMYND10','ZNF10','ZNF100','ZNF101','ZNF106','ZNF107','ZNF112','ZNF114','ZNF117','ZNF12','ZNF121','ZNF124','ZNF131','ZNF132','ZNF133','ZNF134','ZNF135','ZNF136','ZNF137P','ZNF138','ZNF14','ZNF140','ZNF141','ZNF142','ZNF143','ZNF146','ZNF148','ZNF154','ZNF155','ZNF157','ZNF16','ZNF160','ZNF165','ZNF169','ZNF17','ZNF174','ZNF175','ZNF177','ZNF18','ZNF180','ZNF181','ZNF182','ZNF184','ZNF185','ZNF189','ZNF19','ZNF195','ZNF197','ZNF2','ZNF20','ZNF200','ZNF202','ZNF205','ZNF207','ZNF208','ZNF211','ZNF212','ZNF213','ZNF214','ZNF215','ZNF217','ZNF219','ZNF22','ZNF221','ZNF222','ZNF223','ZNF224','ZNF225','ZNF226','ZNF227','ZNF229','ZNF23','ZNF230','ZNF232','ZNF233','ZNF234','ZNF235','ZNF236','ZNF239','ZNF24','ZNF248','ZNF25','ZNF250','ZNF251','ZNF253','ZNF254','ZNF256','ZNF257','ZNF26','ZNF260','ZNF263','ZNF264','ZNF266','ZNF267','ZNF268','ZNF273','ZNF274','ZNF275','ZNF276','ZNF277','ZNF28','ZNF280A','ZNF280B','ZNF280C','ZNF280D','ZNF281','ZNF282','ZNF283','ZNF284','ZNF285','ZNF286A','ZNF286B','ZNF287','ZNF292','ZNF296','ZNF3','ZNF30','ZNF300','ZNF302','ZNF304','ZNF311','ZNF316','ZNF317','ZNF318','ZNF319','ZNF32','ZNF320','ZNF322','ZNF324','ZNF324B','ZNF326','ZNF329','ZNF330','ZNF331','ZNF333','ZNF334','ZNF335','ZNF337','ZNF33A','ZNF33B','ZNF34','ZNF341','ZNF343','ZNF345','ZNF346','ZNF347','ZNF35','ZNF350','ZNF354A','ZNF354B','ZNF354C','ZNF355P','ZNF358','ZNF362','ZNF365','ZNF366','ZNF367','ZNF37A','ZNF382','ZNF383','ZNF384','ZNF385A','ZNF385B','ZNF385C','ZNF385D','ZNF391','ZNF394','ZNF395','ZNF396','ZNF397','ZNF398','ZNF404','ZNF407','ZNF408','ZNF41','ZNF410','ZNF414','ZNF415','ZNF416','ZNF417','ZNF418','ZNF419','ZNF420','ZNF423','ZNF425','ZNF426','ZNF428','ZNF429','ZNF43','ZNF430','ZNF431','ZNF432','ZNF433','ZNF436','ZNF438','ZNF439','ZNF44','ZNF440','ZNF441','ZNF442','ZNF443','ZNF444','ZNF445','ZNF446','ZNF449','ZNF45','ZNF451','ZNF454','ZNF460','ZNF461','ZNF462','ZNF467','ZNF468','ZNF469','ZNF470','ZNF471','ZNF473','ZNF474','ZNF479','ZNF48','ZNF480','ZNF483','ZNF484','ZNF485','ZNF486','ZNF487','ZNF488','ZNF490','ZNF491','ZNF492','ZNF493','ZNF496','ZNF497','ZNF500','ZNF501','ZNF502','ZNF503','ZNF506','ZNF507','ZNF510','ZNF511','ZNF512','ZNF512B','ZNF513','ZNF514','ZNF516','ZNF517','ZNF518A','ZNF518B','ZNF519']
TFs3Other=['ZNF521','ZNF524','ZNF525','ZNF526','ZNF527','ZNF528','ZNF529','ZNF530','ZNF532','ZNF534','ZNF536','ZNF540','ZNF541','ZNF542P','ZNF543','ZNF544','ZNF546','ZNF547','ZNF548','ZNF549','ZNF550','ZNF551','ZNF552','ZNF554','ZNF555','ZNF556','ZNF557','ZNF558','ZNF559','ZNF560','ZNF561','ZNF562','ZNF563','ZNF564','ZNF565','ZNF566','ZNF567','ZNF568','ZNF569','ZNF57','ZNF570','ZNF571','ZNF572','ZNF573','ZNF574','ZNF575','ZNF576','ZNF577','ZNF578','ZNF579','ZNF580','ZNF581','ZNF582','ZNF583','ZNF584','ZNF585A','ZNF585B','ZNF586','ZNF587','ZNF587B','ZNF589','ZNF592','ZNF593','ZNF594','ZNF595','ZNF596','ZNF597','ZNF598','ZNF599','ZNF600','ZNF605','ZNF606','ZNF607','ZNF608','ZNF609','ZNF610','ZNF611','ZNF613','ZNF614','ZNF615','ZNF616','ZNF618','ZNF619','ZNF620','ZNF621','ZNF622','ZNF623','ZNF624','ZNF625','ZNF626','ZNF627','ZNF628','ZNF629','ZNF630','ZNF638','ZNF639','ZNF641','ZNF644','ZNF645','ZNF646','ZNF648','ZNF649','ZNF652','ZNF653','ZNF654','ZNF655','ZNF658','ZNF66','ZNF660','ZNF662','ZNF664','ZNF665','ZNF667','ZNF668','ZNF669','ZNF670','ZNF671','ZNF672','ZNF674','ZNF675','ZNF676','ZNF677','ZNF678','ZNF679','ZNF680','ZNF681','ZNF682','ZNF683','ZNF684','ZNF687','ZNF688','ZNF689','ZNF69','ZNF691','ZNF692','ZNF695','ZNF696','ZNF697','ZNF699','ZNF7','ZNF70','ZNF700','ZNF701','ZNF703','ZNF704','ZNF705A','ZNF705B','ZNF705D','ZNF705E','ZNF705G','ZNF706','ZNF707','ZNF708','ZNF709','ZNF71','ZNF710','ZNF711','ZNF713','ZNF714','ZNF716','ZNF717','ZNF718','ZNF720','ZNF721','ZNF724','ZNF726','ZNF727','ZNF728','ZNF729','ZNF73','ZNF730','ZNF732','ZNF735','ZNF736','ZNF737','ZNF738','ZNF74','ZNF740','ZNF746','ZNF747','ZNF749','ZNF750','ZNF75A','ZNF75D','ZNF76','ZNF761','ZNF763','ZNF764','ZNF765','ZNF766','ZNF768','ZNF77','ZNF770','ZNF771','ZNF772','ZNF773','ZNF774','ZNF775','ZNF776','ZNF777','ZNF778','ZNF780A','ZNF780B','ZNF781','ZNF782','ZNF783','ZNF784','ZNF785','ZNF786','ZNF787','ZNF788','ZNF789','ZNF79','ZNF790','ZNF791','ZNF792','ZNF793','ZNF799','ZNF8','ZNF80','ZNF800','ZNF804A','ZNF804B','ZNF805','ZNF808','ZNF81','ZNF812P','ZNF813','ZNF814','ZNF816','ZNF821','ZNF823','ZNF827','ZNF829','ZNF83','ZNF830','ZNF831','ZNF835','ZNF836','ZNF837','ZNF839','ZNF84','ZNF841','ZNF843','ZNF844','ZNF845','ZNF846','ZNF85','ZNF850','ZNF852','ZNF853','ZNF860','ZNF862','ZNF865','ZNF878','ZNF879','ZNF880','ZNF883','ZNF888','ZNF891','ZNF90','ZNF91','ZNF92','ZNF93','ZNF98','ZNF99','ZNFX1','ZRANB2','ZRSR1','ZRSR2','ZSCAN1','ZSCAN10','ZSCAN12','ZSCAN16','ZSCAN18','ZSCAN2','ZSCAN20','ZSCAN21','ZSCAN22','ZSCAN23','ZSCAN25','ZSCAN26','ZSCAN29','ZSCAN30','ZSCAN31','ZSCAN32','ZSCAN4','ZSCAN5A','ZSCAN5B','ZSCAN5C','ZSCAN5DP','ZSCAN9','ZUFSP','ZXDA','ZXDB','ZXDC','ZZZ3']
TFs4Other=['ATF2','ATF6B','ATN1','BHLHA15','BHLHB2','BHLHB4 ','BHLHB5','BHLHB9','C13ORF8','C20ORF194','CDX2','CRAMP1L','ELP3','ERVK3-1','ETF1','FLJ12895','FLJ20321','FLJ25476','FLJ35784','FOXM1','FOXO2','FOXR1','FOXR2','GATA1','GRLF1','KDM4C','KDM4E','KIAA0863','KIAA0961','KLRG1','LBXCOR1','LEF1','LHX7','LOC126295','LOC51058','LOC91661','MGC4728','MYOD1','MYOG','MYRF','NEUROD4','NEUROG1','NFAT2','NFAT4','NFE2L1','NR1A2','NR1B1','NR1B2','NR1B3','NR1C1','NR1C2','NR1C3','NR1F3','NR1I1','NR2A1','NR2B3','NR3A1','NR3A2','NR3B3','OTEX','PEPP-2','PHF2','POU2F1','POU2F2','POU4F3','POU5F1','POU6F1','PRDM3','PRKRIP1','PRKRIR','SETD1A','SMAD8','SMYD2','SOX12','SOX14','SOX15','TBX14','TCF1','TCF17','TCF2','TCF9','TEAD1','TPRXL','UBTFL1','WHSC1','ZBTB14','ZBTB18','ZBTB19','ZBTB21','ZBTB27','ZBTB28','ZBTB35','ZBTB49','ZFP161','ZFP28','ZKSCAN10','ZKSCAN11','ZKSCAN12','ZKSCAN13','ZKSCAN14','ZKSCAN16','ZKSCAN19','ZKSCAN21','ZKSCAN24','ZKSCAN6','ZKSCAN8','ZNF10','ZNF112','ZNF187','ZNF193','ZNF204P','ZNF238','ZNF271','ZNF295','ZNF305','ZNF323','ZNF434','ZNF499','ZNF542','ZNF545','ZNF553','ZNF56','ZNF625','ZNF643','ZNF698','ZNF705F','ZNF751','ZNF752','ZNF753','ZNF785','ZNF806','ZNF816','ZNF833P','ZNF906','ZSCAN1','ZSCAN25','ZSCAN26','ZSCAN31','ZSCAN32','ZSCAN5D','ZSCAN9']
TFsOther=TFs1Other+TFs2Other+TFs3Other+TFs4Other
```

Example 5 (python):
```python
# preserve raw count
adata.layers["counts"] = adata.X.copy()

# normalise and log-transform
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5, batch_key='batch_key')

print(sum(adata.var.highly_variable))
sc.pl.highly_variable_genes(adata)

# freeze the state in `.raw`
adata.raw = adata

# subset hvgs
adata = adata[:, adata.var.highly_variable]

adata=adata.copy()

sc.tl.score_genes_cell_cycle(adata, s_genes=s_genes, g2m_genes=g2m_genes)
#sc.pp.regress_out(adata, ['S_score','G2M_score'])
```

Example 6 (python):
```python
#batch correction
scvi.model.SCVI.setup_anndata(adata, 
                              layer="counts", 
                              batch_key="batch_key",
                              categorical_covariate_keys=['phase'],
                              continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo'])

vae = scvi.model.SCVI(adata, n_hidden = 128, n_latent = 50, n_layers = 3, dispersion = 'gene-batch')
vae.train(max_epochs=50)
    
# Convergence plot
x = np.linspace(1, (len(vae.history['elbo_train'])), (len(vae.history['elbo_train'])))
plt.plot(x, vae.history['elbo_train'], label="train")
plt.xlabel('epoch')
plt.ylabel('elbo_train')
plt.show()
plt.clf()
```

Example 7 (python):
```python
# add the latent space to adata
adata.obsm["X_scVI"] = vae.get_latent_representation()

sc.tl.score_genes(adata, stress_genes, score_name='stress_score')
sc.tl.score_genes(adata, hb_genes1, score_name='hb1_score')

sc.pp.neighbors(adata, use_rep="X_scVI")
sc.tl.umap(adata,min_dist=0.1)
#sc.tl.tsne(adata)
```

Example 8 (python):
```python
sc.pl.umap(adata, color=['log1p_total_counts','n_genes','n_genes_by_counts','total_counts','pct_counts_mt','pct_counts_ribo'],ncols=5)
sc.pl.umap(adata, color=['scrublet_score','stress_score', 'hb1_score','phase'])
sc.pl.umap(adata, color=['donor','age','region','phase','cell_or_nuclei','modality','kit_10x'],wspace=0.8)
Genes=['TTN','TNNT2','MYOM1','LDB3','ALDH1A2','TNNT1','KRT19','UPK3B','BNC1','COL6A3','FBLN1','PCOLCE','LRRC17','PCDH18','EMCN','ECSCR','PECAM1','CDH5','ESAM', 'LAPTM5','PTPRC','LCP1','CD53','WAS','PHOX2B','NRXN1','STMN2','GRIK2','L1CAM']
sc.pl.umap(adata, color=Genes)
```

Example 9 (python):
```python
sc.pl.umap(adata, color=['LYVE1','PECAM1'])
```

Example 10 (python):
```python
for k in list(set(adata.obs.sangerID)):
    sc.pl.umap(adata, color='sangerID',groups=k)
```

Example 11 (python):
```python
for k in list(set(adata.obs.region)):
    sc.pl.umap(adata, color='region',groups=k)
```

Example 12 (python):
```python
sc.tl.leiden(adata, resolution=1, key_added='leiden')
sc.pl.umap(adata, color='leiden',legend_loc='on data')
```

Example 13 (python):
```python
CardiomyocyteClusters=['1','15','28','20','5','11','3']
MesenchymalClusters=['2','23','10','0','12','6','33','14','27','29','4','17','22','24','26','17','13','16','31']
EndotheliumClusters=['8','35','7','25']
EpicardiumClusters=['19']
ImmuneClusters=['9','18','37','21','34','38','36']
NeuralClusters=['32','30']
adata.obs['coarse_grain']= 'na'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in CardiomyocyteClusters]]='Cardiomyocytes'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in MesenchymalClusters]]='Mesenchymal'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in EndotheliumClusters]]='Endothelium'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in EpicardiumClusters]]='Epicardium'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in ImmuneClusters]]='Immune'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in NeuralClusters]]='Neural'
sc.pl.umap(adata, color='coarse_grain')
```

Example 14 (python):
```python
#Ito cells
sc.pl.umap(adata, color=['OIT3','CRHBP','FCN3'])

#Parathyroid cells
sc.pl.umap(adata, color=['PTH','MAFB','GATA3'])

#Spleen
sc.pl.umap(adata, color=['VPREB3','MZB1','CD79B'])

#Hepatocytes
sc.pl.umap(adata, color=['ALB','APOA1','TTR'])

#Melanocytes
sc.pl.umap(adata, color=['MLANA','DCT','DCD','ALX1','CCER2'])

#Adipocytes
sc.pl.umap(adata, color=['GPAM','FASN','ADIPOQ'])

#Tenocytes
sc.pl.umap(adata, color=['TNMD','SCX','TNC','CNMD','THBS4'])

#Chondrocyte markers
sc.pl.umap(adata, color=['MATN4','ACAN','CHAD','COL6A3','COL9A1','COL9A3','HAPLN1','CYTL1','COL2A1','COMP','MKX','SOX9','CD74','EFEMP1','TGM2'])

#AT/12 cells
sc.pl.umap(adata, color=['EPCAM','HOPX','LAMP3','SFTPC'])

#Neural crest cells
sc.pl.umap(adata, color=['FOXD3','SNAI1','SNAI2','SOX8','SOX9','SOX10','TFAP2A'])

#Placenta cytotrophoblasts
sc.pl.umap(adata, color=['PAGE4','GSTA3'])

#Esophagus epithelial
sc.pl.umap(adata, color=['SOX2','TP63','TRH'])
```

Example 15 (python):
```python
adata.write('./Objects/T21_multiome_RNA_foetal_scVIProcessed.h5ad')
```

Example 16 (python):
```python
adata=sc.read('./Objects/T21_multiome_RNA_foetal.h5ad')
print(adata.shape)

bdata=sc.read('./Objects/T21_multiome_RNA_foetal_scVIProcessed.h5ad')
print(bdata.shape)
#Select Cardiomyocyte, Mesenchymal, Endothelium, Epicardium, Immune, Neural
bdata=bdata[bdata.obs.coarse_grain=='Immune',:].copy()

adata=adata[[x for x in adata.obs_names if x in bdata.obs_names],:].copy()
print(adata.shape)
del bdata

# set gene name as var_names 
adata.var['gene_id']=adata.var_names.copy()
adata.var.set_index('gene_name',inplace=True)
adata.var_names=adata.var_names.astype('str')
adata.var_names_make_unique()

cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP','RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2','USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67','TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1','CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF','NEK2','G2E3','GAS2L3','CBX5','CENPA']
s_genes = cell_cycle_genes[:43]
s_genes = [x for x in s_genes if x in adata.var_names]
print('S genes:', s_genes, '\n')
g2m_genes = cell_cycle_genes[43:]
g2m_genes = [x for x in g2m_genes if x in adata.var_names]
print('G2 genes:', g2m_genes, '\n')
cell_cycle_genes = [x for x in cell_cycle_genes if x in adata.var_names]

hb_genes1 = ['HBB','HBD','HBG1','HBG2','HBE1','HBZ','HBM','HBA2','HBA1', 'HBQ1']
hb_genes1 = [x for x in hb_genes1 if x in adata.var_names]
print('Hemoglobin genes set 1:', hb_genes1, '\n')

hb_genes2 = ['AHSP','GYPA','RHAG','GFI1B','KEL','EPB42','ALAS2','TSPO2','GYPB','GYPE','SPTA1','RHCE','KLF1','RHD','SLC4A1','SNX22','SQSTM1','SOD1','FTH1','MAP1LC3B','SLC25A37','TRIM58','HMBS']
hb_genes2 = [x for x in hb_genes2 if x in adata.var_names]
print('Hemoglobin genes set 2:', hb_genes2, '\n')

mito_genes=['MT-ND1','MT-ND2','MT-CO1','MT-CO2','MT-ATP8','MT-ATP6','MT-CO3','MT-ND3','MT-ND4L', 'MT-ND4', 'MT-ND5', 'MT-ND6', 'MT-CYB']
mito_genes = [x for x in mito_genes if x in adata.var_names]
print('Mitochondrial genes:', mito_genes, '\n')

ribo_genes = ['RPLP0','RPLP1','RPLP2','RPL3','RPL3L','RPL4','RPL5','RPL6','RPL7','RPL7A','RPL7L1','RPL8','RPL9','RPL10','RPL10A','RPL10L','RPL11','RPL12','RPL13','RPL13A','RPL14','RPL15','RPL17','RPL18','RPL18A','RPL19','RPL21','RPL22','RPL22L1','RPL23','RPL23A','RPL24','RPL26','RPL26L1','RPL27','RPL27A','RPL28','RPL29','RPL30','RPL31','RPL32','RPL34','RPL35','RPL35A','RPL36','RPL36A','RPL36AL','RPL37','RPL37A','RPL38','RPL39','RPL39L','UBA52','RPL41','RPSA','RPS2','RPS3','RPS3A','RPS4X','RPS4Y1','RPS4Y2','RPS5','RPS6','RPS7','RPS8','RPS9','RPS10','RPS11','RPS12','RPS13','RPS14','RPS15','RPS15A','RPS16','RPS17','RPS18','RPS19','RPS20','RPS21','RPS23','RPS24','RPS25','RPS26','RPS27','RPS27A','RPS27L','RPS28','RPS29','FAU']
ribo_genes = [x for x in ribo_genes if x in adata.var_names]
print('Ribo genes:', ribo_genes, '\n')

stress_genes=['FOS','FOSB','JUN','JUNB','JUND','ZFP36','SOCS3','CEBPD','CEBPB','ATF3','EGR1','HSPA1A','HSPA1B','HSP90AA1','HSP90AB1','HSPA8','HSPB1','HSPE1','IER3','IER2','BTG1','BTG2','DUSP1']
stress_genes = [x for x in stress_genes if x in adata.var_names]
print('Tissue dissociation stress associated genes:', stress_genes, '\n')

# preserve raw count
adata.layers["counts"] = adata.X.copy()

# normalise and log-transform
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5, batch_key='batch_key')

print(sum(adata.var.highly_variable))
sc.pl.highly_variable_genes(adata)

# freeze the state in `.raw`
adata.raw = adata

# subset hvgs
adata = adata[:, adata.var.highly_variable]

adata=adata.copy()

sc.tl.score_genes_cell_cycle(adata, s_genes=s_genes, g2m_genes=g2m_genes)
#sc.pp.regress_out(adata, ['S_score','G2M_score'])

#batch correction
scvi.model.SCVI.setup_anndata(adata, 
                              layer="counts", 
                              batch_key="batch_key",
                              categorical_covariate_keys=['phase'],
                              continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo'])

vae = scvi.model.SCVI(adata, n_hidden = 128, n_latent = 50, n_layers = 3, dispersion = 'gene-batch')
vae.train(max_epochs=50)
    
# Convergence plot
x = np.linspace(1, (len(vae.history['elbo_train'])), (len(vae.history['elbo_train'])))
plt.plot(x, vae.history['elbo_train'], label="train")
plt.xlabel('epoch')
plt.ylabel('elbo_train')
plt.show()
plt.clf()

# add the latent space to adata
adata.obsm["X_scVI"] = vae.get_latent_representation()

#sc.tl.score_genes(adata, stress_genes, score_name='stress_score')

sc.tl.score_genes(adata, hb_genes1, score_name='hb1_score')

sc.pp.neighbors(adata, use_rep="X_scVI")
sc.tl.umap(adata,min_dist=0.1)
#sc.tl.tsne(adata)

sc.pl.umap(adata, color=['log1p_total_counts','n_genes','n_genes_by_counts','total_counts','pct_counts_mt','pct_counts_ribo'],ncols=5)
sc.pl.umap(adata, color=['scrublet_score', 'hb1_score','phase'])
sc.pl.umap(adata, color=['donor','age','region','phase','cell_or_nuclei','modality','kit_10x'],wspace=0.8)
```

Example 17 (python):
```python
sc.pl.umap(adata, color=hb_genes1)
sc.pl.umap(adata, color=hb_genes2)
```

Example 18 (python):
```python
Genes=['TTN','TNNT2','MYOM1','LDB3','ALDH1A2','TNNT1','KRT19','UPK3B','BNC1','COL6A3','FBLN1','PCOLCE','LRRC17','PCDH18','EMCN','ECSCR','PECAM1','CDH5','ESAM', 'LAPTM5','PTPRC','LCP1','CD53','WAS','PHOX2B','NRXN1','STMN2','GRIK2','L1CAM']
sc.pl.umap(adata, color=Genes)

#Ito cells
sc.pl.umap(adata, color=['OIT3','CRHBP','FCN3'])

#Parathyroid cells
sc.pl.umap(adata, color=['PTH','MAFB','GATA3'])

#Spleen
sc.pl.umap(adata, color=['VPREB3','MZB1','CD79B'])

#Hepatocytes
sc.pl.umap(adata, color=['ALB','APOA1','TTR'])

#Melanocytes
sc.pl.umap(adata, color=['MLANA','DCT','DCD','ALX1','CCER2'])

#Adipocytes
sc.pl.umap(adata, color=['GPAM','FASN','ADIPOQ'])

#Tenocytes
sc.pl.umap(adata, color=['TNMD','SCX','TNC','CNMD','THBS4'])

#Chondrocyte markers
sc.pl.umap(adata, color=['MATN4','ACAN','CHAD','COL6A3','COL9A1','COL9A3','HAPLN1','CYTL1','COL2A1','COMP','MKX','SOX9','CD74','EFEMP1','TGM2'])

#AT/12 cells
sc.pl.umap(adata, color=['EPCAM','HOPX','LAMP3','SFTPC'])

#Neural crest cells
sc.pl.umap(adata, color=['FOXD3','SNAI1','SNAI2','SOX8','SOX9','SOX10','TFAP2A'])

#Placenta cytotrophoblasts
sc.pl.umap(adata, color=['PAGE4','GSTA3'])

#Esophagus epithelial
sc.pl.umap(adata, color=['SOX2','TP63','TRH'])
```

Example 19 (python):
```python
sc.tl.leiden(adata, resolution=0.75, key_added='leiden')  #Cluster 11 CMs, cluster 19 RBCs
sc.pl.umap(adata, color='leiden',legend_loc='on data')
```

Example 20 (python):
```python
sc.pl.dotplot(adata, ['PTPRC','MYL7','NPPA','TNNT2'] ,groupby='leiden',standard_scale='var',cmap='PRGn')
sc.pl.umap(adata, color=['PTPRC','MYL7','NPPA','TNNT2'])
```

Example 21 (python):
```python
#this block is to see where the cells over a certain threshold with TNNT2 expression lies in the UMAP
#enter what gene to set threshold for
#when running this box for the first time, comment the starting with 'del'
#then uncomment it for the other ones --- !!!

sc.pl.umap(adata, color=['TNNT2'])
CellName='Cardio_toremove'
gene='TNNT2'
exp_thresh=1.5
gene_pos=adata.var_names.get_loc(gene)
adata_sel = adata[adata.X[:,gene_pos]>exp_thresh].copy()

l=adata_sel.obs_names
set(l).intersection(adata.obs_names)
print(len(l))

#del adata.obs[CellName]
adata.obs.loc[set(l).intersection(adata.obs_names),CellName]='Yes'
sc.settings.set_figure_params(dpi=200, facecolor='white', color_map = 'RdPu',)
sc.pl.umap(adata, color=[CellName], groups='Yes',size=10, wspace=0.45, legend_fontsize=5, legend_loc='on data',frameon=False)
```

Example 22 (python):
```python
bdata = adata[adata.obs['leiden'].isin(['11','19'])].copy()
bdatanames=bdata.obs_names #this is for the obvious clusters to remove directly
cdatanames=adata_sel.obs_names #this is from the removed cells above

p=set(cdatanames)
r=set(bdatanames)
p.update(r)
p=list(p)

print(len(p))
np.savetxt('./Trisomy/RemovedCells/RBCandCardioRemovedFromImmune.txt', p, delimiter="\n", fmt="%s")
```

Example 23 (python):
```python
adata=sc.read('./Objects/T21_multiome_RNA_foetal.h5ad')
print(adata.shape)

bdata=sc.read('./Objects/T21_multiome_RNA_foetal_scVIProcessed.h5ad')
print(bdata.shape)
#Select Cardiomyocyte, Mesenchymal, Endothelium, Epicardium, Immune, Neural
bdata=bdata[bdata.obs.coarse_grain=='Epicardium',:].copy()

adata=adata[[x for x in adata.obs_names if x in bdata.obs_names],:].copy()
print(adata.shape)
del bdata

# set gene name as var_names 
adata.var['gene_id']=adata.var_names.copy()
adata.var.set_index('gene_name',inplace=True)
adata.var_names=adata.var_names.astype('str')
adata.var_names_make_unique()

cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP','RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2','USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67','TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1','CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF','NEK2','G2E3','GAS2L3','CBX5','CENPA']
s_genes = cell_cycle_genes[:43]
s_genes = [x for x in s_genes if x in adata.var_names]
print('S genes:', s_genes, '\n')
g2m_genes = cell_cycle_genes[43:]
g2m_genes = [x for x in g2m_genes if x in adata.var_names]
print('G2 genes:', g2m_genes, '\n')
cell_cycle_genes = [x for x in cell_cycle_genes if x in adata.var_names]

hb_genes1 = ['HBB','HBD','HBG1','HBG2','HBE1','HBZ','HBM','HBA2','HBA1', 'HBQ1']
hb_genes1 = [x for x in hb_genes1 if x in adata.var_names]
print('Hemoglobin genes set 1:', hb_genes1, '\n')

hb_genes2 = ['AHSP','GYPA','RHAG','GFI1B','KEL','EPB42','ALAS2','TSPO2','GYPB','GYPE','SPTA1','RHCE','KLF1','RHD','SLC4A1','SNX22','SQSTM1','SOD1','FTH1','MAP1LC3B','SLC25A37','TRIM58','HMBS']
hb_genes2 = [x for x in hb_genes2 if x in adata.var_names]
print('Hemoglobin genes set 2:', hb_genes2, '\n')

mito_genes=['MT-ND1','MT-ND2','MT-CO1','MT-CO2','MT-ATP8','MT-ATP6','MT-CO3','MT-ND3','MT-ND4L', 'MT-ND4', 'MT-ND5', 'MT-ND6', 'MT-CYB']
mito_genes = [x for x in mito_genes if x in adata.var_names]
print('Mitochondrial genes:', mito_genes, '\n')

ribo_genes = ['RPLP0','RPLP1','RPLP2','RPL3','RPL3L','RPL4','RPL5','RPL6','RPL7','RPL7A','RPL7L1','RPL8','RPL9','RPL10','RPL10A','RPL10L','RPL11','RPL12','RPL13','RPL13A','RPL14','RPL15','RPL17','RPL18','RPL18A','RPL19','RPL21','RPL22','RPL22L1','RPL23','RPL23A','RPL24','RPL26','RPL26L1','RPL27','RPL27A','RPL28','RPL29','RPL30','RPL31','RPL32','RPL34','RPL35','RPL35A','RPL36','RPL36A','RPL36AL','RPL37','RPL37A','RPL38','RPL39','RPL39L','UBA52','RPL41','RPSA','RPS2','RPS3','RPS3A','RPS4X','RPS4Y1','RPS4Y2','RPS5','RPS6','RPS7','RPS8','RPS9','RPS10','RPS11','RPS12','RPS13','RPS14','RPS15','RPS15A','RPS16','RPS17','RPS18','RPS19','RPS20','RPS21','RPS23','RPS24','RPS25','RPS26','RPS27','RPS27A','RPS27L','RPS28','RPS29','FAU']
ribo_genes = [x for x in ribo_genes if x in adata.var_names]
print('Ribo genes:', ribo_genes, '\n')

stress_genes=['FOS','FOSB','JUN','JUNB','JUND','ZFP36','SOCS3','CEBPD','CEBPB','ATF3','EGR1','HSPA1A','HSPA1B','HSP90AA1','HSP90AB1','HSPA8','HSPB1','HSPE1','IER3','IER2','BTG1','BTG2','DUSP1']
stress_genes = [x for x in stress_genes if x in adata.var_names]
print('Tissue dissociation stress associated genes:', stress_genes, '\n')

# preserve raw count
adata.layers["counts"] = adata.X.copy()

# normalise and log-transform
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5, batch_key='batch_key')

print(sum(adata.var.highly_variable))
sc.pl.highly_variable_genes(adata)

# freeze the state in `.raw`
adata.raw = adata

# subset hvgs
adata = adata[:, adata.var.highly_variable]

adata=adata.copy()

sc.tl.score_genes_cell_cycle(adata, s_genes=s_genes, g2m_genes=g2m_genes)
#sc.pp.regress_out(adata, ['S_score','G2M_score'])

#batch correction
scvi.model.SCVI.setup_anndata(adata, 
                              layer="counts", 
                              batch_key="batch_key",
                              categorical_covariate_keys=['phase'],
                              continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo'])

vae = scvi.model.SCVI(adata, n_hidden = 128, n_latent = 50, n_layers = 3, dispersion = 'gene-batch')
vae.train(max_epochs=100)
    
# Convergence plot
x = np.linspace(1, (len(vae.history['elbo_train'])), (len(vae.history['elbo_train'])))
plt.plot(x, vae.history['elbo_train'], label="train")
plt.xlabel('epoch')
plt.ylabel('elbo_train')
plt.show()
plt.clf()

# add the latent space to adata
adata.obsm["X_scVI"] = vae.get_latent_representation()

#sc.tl.score_genes(adata, stress_genes, score_name='stress_score')

sc.tl.score_genes(adata, hb_genes1, score_name='hb1_score')

sc.pp.neighbors(adata, use_rep="X_scVI")
sc.tl.umap(adata,min_dist=0.1)
#sc.tl.tsne(adata)

sc.pl.umap(adata, color=['log1p_total_counts','n_genes','n_genes_by_counts','total_counts','pct_counts_mt','pct_counts_ribo'],ncols=5)
sc.pl.umap(adata, color=['scrublet_score', 'hb1_score','phase'])
sc.pl.umap(adata, color=['donor','age','region','phase','cell_or_nuclei','modality','kit_10x'],wspace=0.8)
```

Example 24 (python):
```python
sc.pl.umap(adata, color=['scrublet_score', 'hb1_score','phase'])
sc.pl.umap(adata, color=['donor','age','region','phase','cell_or_nuclei','modality','kit_10x'],wspace=0.8)
```

Example 25 (python):
```python
#Hepatocyte
sc.pl.umap(adata, color=['ALB','APOA1','TTR'])
```

Example 26 (python):
```python
#Epi markers
sc.pl.umap(adata, color=['MSLN','TCF21','POSTN','ITLN1','DCN','TBX18','BNC1','GPC6'])


sc.pl.umap(adata, color=hb_genes1)
sc.pl.umap(adata, color=hb_genes2)

Genes=['TTN','TNNT2','MYOM1','LDB3','ALDH1A2','TNNT1','KRT19','UPK3B','BNC1','COL6A3','FBLN1','PCOLCE','LRRC17','PCDH18','EMCN','ECSCR','PECAM1','CDH5','ESAM', 'LAPTM5','PTPRC','LCP1','CD53','WAS','PHOX2B','NRXN1','STMN2','GRIK2','L1CAM']
sc.pl.umap(adata, color=Genes)
```

Example 27 (python):
```python
sc.pl.umap(adata, color='hb1_score')
```

Example 28 (python):
```python
sc.pl.umap(adata, color=hb_genes1)
sc.pl.umap(adata, color=hb_genes2)

for k in list(set(adata.obs.region)):
    sc.pl.umap(adata, color='region',groups=k)
    
for k in list(set(adata.obs.sangerID)):
    sc.pl.umap(adata, color='sangerID',groups=k)
```

Example 29 (python):
```python
#Ito cells
sc.pl.umap(adata, color=['OIT3','CRHBP','FCN3'])

#Parathyroid cells
sc.pl.umap(adata, color=['PTH','MAFB','GATA3'])

#Spleen
sc.pl.umap(adata, color=['VPREB3','MZB1','CD79B'])

#Hepatocytes
sc.pl.umap(adata, color=['ALB','APOA1','TTR'])

#Melanocytes
sc.pl.umap(adata, color=['MLANA','DCT','DCD','ALX1','CCER2'])

#Adipocytes
sc.pl.umap(adata, color=['GPAM','FASN','ADIPOQ'])

#Tenocytes
sc.pl.umap(adata, color=['TNMD','SCX','TNC','CNMD','THBS4'])

#Chondrocyte markers
sc.pl.umap(adata, color=['MATN4','ACAN','CHAD','COL6A3','COL9A1','COL9A3','HAPLN1','CYTL1','COL2A1','COMP','MKX','SOX9','CD74','EFEMP1','TGM2'])

#AT/12 cells
sc.pl.umap(adata, color=['EPCAM','HOPX','LAMP3','SFTPC'])

#Neural crest cells
sc.pl.umap(adata, color=['FOXD3','SNAI1','SNAI2','SOX8','SOX9','SOX10','TFAP2A'])

#Placenta cytotrophoblasts
sc.pl.umap(adata, color=['PAGE4','GSTA3'])

#Esophagus epithelial
sc.pl.umap(adata, color=['SOX2','TP63','TRH'])
```

Example 30 (python):
```python
sc.tl.leiden(adata, resolution=0.05, key_added='leiden') #cluster 12 is RBCs, cluster 13 and 7 are CMs
sc.pl.umap(adata, color='leiden',legend_loc='on data')

ClusterOfInterest='2'
sc.tl.rank_genes_groups(adata, 'leiden', method='t-test', key_added = "t-test")
DEGenesForCluster = [x[ClusterOfInterest] for x in adata.uns['t-test']['names'][:50]]
sc.pl.dotplot(adata, DEGenesForCluster ,groupby='leiden',standard_scale='var',cmap='PRGn')
#sc.pl.umap(adata, color=DEGenesForCluster)
```

Example 31 (python):
```python
#this block is to see where the cells over a certain threshold with TNNT2 expression lies in the UMAP
#enter what gene to set threshold for
#when running this box for the first time, comment the starting with 'del'
#then uncomment it for the other ones --- !!!

sc.pl.umap(adata, color=['TNNT2'])
CellName='Cardio_toremove'
gene='TNNT2'
exp_thresh=1.3
gene_pos=adata.var_names.get_loc(gene)
adata_sel = adata[adata.X[:,gene_pos]>exp_thresh].copy()

l=adata_sel.obs_names
set(l).intersection(adata.obs_names)
print(len(l))

del adata.obs[CellName]
adata.obs.loc[set(l).intersection(adata.obs_names),CellName]='Yes'
sc.settings.set_figure_params(dpi=200, facecolor='white', color_map = 'RdPu',)
sc.pl.umap(adata, color=[CellName], groups='Yes',size=10, wspace=0.45, legend_fontsize=5, legend_loc='on data',frameon=False)
```

Example 32 (python):
```python
bdatanames=adata_sel.obs_names #this is for the obvious clusters to remove directly


print(len(bdatanames))
np.savetxt('./Trisomy/RemovedCells/RBCandCardioRemovedFromEpi.txt', bdatanames, delimiter="\n", fmt="%s")
```

Example 33 (python):
```python
adata=sc.read('./Objects/T21_multiome_RNA_foetal.h5ad')
print(adata.shape)

bdata=sc.read('./Objects/T21_multiome_RNA_foetal_scVIProcessed.h5ad')
print(bdata.shape)
#Select Cardiomyocyte, Mesenchymal, Endothelium, Epicardium, Immune, Neural
bdata=bdata[bdata.obs.coarse_grain=='Endothelium',:].copy()

adata=adata[[x for x in adata.obs_names if x in bdata.obs_names],:].copy()
print(adata.shape)
del bdata

# set gene name as var_names 
adata.var['gene_id']=adata.var_names.copy()
adata.var.set_index('gene_name',inplace=True)
adata.var_names=adata.var_names.astype('str')
adata.var_names_make_unique()

cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP','RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2','USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67','TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1','CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF','NEK2','G2E3','GAS2L3','CBX5','CENPA']
s_genes = cell_cycle_genes[:43]
s_genes = [x for x in s_genes if x in adata.var_names]
print('S genes:', s_genes, '\n')
g2m_genes = cell_cycle_genes[43:]
g2m_genes = [x for x in g2m_genes if x in adata.var_names]
print('G2 genes:', g2m_genes, '\n')
cell_cycle_genes = [x for x in cell_cycle_genes if x in adata.var_names]

hb_genes1 = ['HBB','HBD','HBG1','HBG2','HBE1','HBZ','HBM','HBA2','HBA1', 'HBQ1']
hb_genes1 = [x for x in hb_genes1 if x in adata.var_names]
print('Hemoglobin genes set 1:', hb_genes1, '\n')

hb_genes2 = ['AHSP','GYPA','RHAG','GFI1B','KEL','EPB42','ALAS2','TSPO2','GYPB','GYPE','SPTA1','RHCE','KLF1','RHD','SLC4A1','SNX22','SQSTM1','SOD1','FTH1','MAP1LC3B','SLC25A37','TRIM58','HMBS']
hb_genes2 = [x for x in hb_genes2 if x in adata.var_names]
print('Hemoglobin genes set 2:', hb_genes2, '\n')

mito_genes=['MT-ND1','MT-ND2','MT-CO1','MT-CO2','MT-ATP8','MT-ATP6','MT-CO3','MT-ND3','MT-ND4L', 'MT-ND4', 'MT-ND5', 'MT-ND6', 'MT-CYB']
mito_genes = [x for x in mito_genes if x in adata.var_names]
print('Mitochondrial genes:', mito_genes, '\n')

ribo_genes = ['RPLP0','RPLP1','RPLP2','RPL3','RPL3L','RPL4','RPL5','RPL6','RPL7','RPL7A','RPL7L1','RPL8','RPL9','RPL10','RPL10A','RPL10L','RPL11','RPL12','RPL13','RPL13A','RPL14','RPL15','RPL17','RPL18','RPL18A','RPL19','RPL21','RPL22','RPL22L1','RPL23','RPL23A','RPL24','RPL26','RPL26L1','RPL27','RPL27A','RPL28','RPL29','RPL30','RPL31','RPL32','RPL34','RPL35','RPL35A','RPL36','RPL36A','RPL36AL','RPL37','RPL37A','RPL38','RPL39','RPL39L','UBA52','RPL41','RPSA','RPS2','RPS3','RPS3A','RPS4X','RPS4Y1','RPS4Y2','RPS5','RPS6','RPS7','RPS8','RPS9','RPS10','RPS11','RPS12','RPS13','RPS14','RPS15','RPS15A','RPS16','RPS17','RPS18','RPS19','RPS20','RPS21','RPS23','RPS24','RPS25','RPS26','RPS27','RPS27A','RPS27L','RPS28','RPS29','FAU']
ribo_genes = [x for x in ribo_genes if x in adata.var_names]
print('Ribo genes:', ribo_genes, '\n')

stress_genes=['FOS','FOSB','JUN','JUNB','JUND','ZFP36','SOCS3','CEBPD','CEBPB','ATF3','EGR1','HSPA1A','HSPA1B','HSP90AA1','HSP90AB1','HSPA8','HSPB1','HSPE1','IER3','IER2','BTG1','BTG2','DUSP1']
stress_genes = [x for x in stress_genes if x in adata.var_names]
print('Tissue dissociation stress associated genes:', stress_genes, '\n')

# preserve raw count
adata.layers["counts"] = adata.X.copy()

# normalise and log-transform
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5, batch_key='batch_key')

print(sum(adata.var.highly_variable))
sc.pl.highly_variable_genes(adata)

# freeze the state in `.raw`
adata.raw = adata

# subset hvgs
adata = adata[:, adata.var.highly_variable]

adata=adata.copy()

sc.tl.score_genes_cell_cycle(adata, s_genes=s_genes, g2m_genes=g2m_genes)
#sc.pp.regress_out(adata, ['S_score','G2M_score'])

#batch correction
scvi.model.SCVI.setup_anndata(adata, 
                              layer="counts", 
                              batch_key="batch_key",
                              categorical_covariate_keys=['phase'],
                              continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo'])

vae = scvi.model.SCVI(adata, n_hidden = 128, n_latent = 50, n_layers = 3, dispersion = 'gene-batch')
vae.train(max_epochs=50)
    
# Convergence plot
x = np.linspace(1, (len(vae.history['elbo_train'])), (len(vae.history['elbo_train'])))
plt.plot(x, vae.history['elbo_train'], label="train")
plt.xlabel('epoch')
plt.ylabel('elbo_train')
plt.show()
plt.clf()

# add the latent space to adata
adata.obsm["X_scVI"] = vae.get_latent_representation()

#sc.tl.score_genes(adata, stress_genes, score_name='stress_score')

sc.tl.score_genes(adata, hb_genes1, score_name='hb1_score')

sc.pp.neighbors(adata, use_rep="X_scVI")
sc.tl.umap(adata,min_dist=0.1)
#sc.tl.tsne(adata)

sc.pl.umap(adata, color=['log1p_total_counts','n_genes','n_genes_by_counts','total_counts','pct_counts_mt','pct_counts_ribo'],ncols=5)
sc.pl.umap(adata, color=['scrublet_score', 'hb1_score','phase'])
sc.pl.umap(adata, color=['donor','age','region','phase','cell_or_nuclei','modality','kit_10x'],wspace=0.8)
```

Example 34 (python):
```python
Genes=['TTN','TNNT2','MYOM1','LDB3','ALDH1A2','TNNT1','KRT19','UPK3B','BNC1','COL6A3','FBLN1','PCOLCE','LRRC17','PCDH18','EMCN','ECSCR','PECAM1','CDH5','ESAM', 'LAPTM5','PTPRC','LCP1','CD53','WAS','PHOX2B','NRXN1','STMN2','GRIK2','L1CAM']
sc.pl.umap(adata, color=Genes)

#Ito cells
sc.pl.umap(adata, color=['OIT3','CRHBP','FCN3'])

#Parathyroid cells
sc.pl.umap(adata, color=['PTH','MAFB','GATA3'])

#Spleen
sc.pl.umap(adata, color=['VPREB3','MZB1','CD79B'])

#Hepatocytes
sc.pl.umap(adata, color=['ALB','APOA1','TTR'])

#Melanocytes
sc.pl.umap(adata, color=['MLANA','DCT','DCD','ALX1','CCER2'])

#Adipocytes
sc.pl.umap(adata, color=['GPAM','FASN','ADIPOQ'])

#Tenocytes
sc.pl.umap(adata, color=['TNMD','SCX','TNC','CNMD','THBS4'])

#Chondrocyte markers
sc.pl.umap(adata, color=['MATN4','ACAN','CHAD','COL6A3','COL9A1','COL9A3','HAPLN1','CYTL1','COL2A1','COMP','MKX','SOX9','CD74','EFEMP1','TGM2'])

#AT/12 cells
sc.pl.umap(adata, color=['EPCAM','HOPX','LAMP3','SFTPC'])

#Neural crest cells
sc.pl.umap(adata, color=['FOXD3','SNAI1','SNAI2','SOX8','SOX9','SOX10','TFAP2A'])

#Placenta cytotrophoblasts
sc.pl.umap(adata, color=['PAGE4','GSTA3'])

#Esophagus epithelial
sc.pl.umap(adata, color=['SOX2','TP63','TRH'])
```

Example 35 (python):
```python
sc.pl.umap(adata, color=['LYVE1','NPR3','NFATC1','FBLN5','RGCC','HAS2','ASPN','KCNJ2'])
```

Example 36 (python):
```python
sc.pl.umap(adata, color=['MYOM1'])
```

Example 37 (python):
```python
sc.tl.leiden(adata, resolution=0.4, key_added='leiden') #Cluster 14 hepatocytes, cluster 10 and 6 cardios
sc.pl.umap(adata, color='leiden',legend_loc='on data')
```

Example 38 (python):
```python
#this block is to see where the cells over a certain threshold with TNNT2 expression lies in the UMAP
#enter what gene to set threshold for
#when running this box for the first time, comment the starting with 'del'
#then uncomment it for the other ones --- !!!

sc.pl.umap(adata, color=['TNNT2'])
CellName='Cardio_toremove'
gene='TNNT2'
exp_thresh=2
gene_pos=adata.var_names.get_loc(gene)
adata_sel = adata[adata.X[:,gene_pos]>exp_thresh].copy()

l=adata_sel.obs_names
set(l).intersection(adata.obs_names)
print(len(l))

#del adata.obs[CellName]
adata.obs.loc[set(l).intersection(adata.obs_names),CellName]='Yes'
sc.settings.set_figure_params(dpi=200, facecolor='white', color_map = 'RdPu',)
sc.pl.umap(adata, color=[CellName], groups='Yes',size=10, wspace=0.45, legend_fontsize=5, legend_loc='on data',frameon=False)
```

Example 39 (python):
```python
bdata = adata[adata.obs['leiden'].isin(['14','6','10'])].copy()
bdatanames=bdata.obs_names #this is for the obvious clusters to remove directly
cdatanames=adata_sel.obs_names #this is from the removed cells above

p=set(cdatanames)
r=set(bdatanames)
p.update(r)
p=list(p)

print(len(p))
np.savetxt('./Trisomy/RemovedCells/RBCandCardioRemovedFromEndo.txt', p, delimiter="\n", fmt="%s")
```

Example 40 (python):
```python
adata=sc.read('./Objects/T21_multiome_RNA_foetal.h5ad')
print(adata.shape)

bdata=sc.read('./Objects/T21_multiome_RNA_foetal_scVIProcessed.h5ad')
print(bdata.shape)
#Select Cardiomyocyte, Mesenchymal, Endothelium, Epicardium, Immune, Neural
bdata=bdata[bdata.obs.coarse_grain=='Neural',:].copy()

adata=adata[[x for x in adata.obs_names if x in bdata.obs_names],:].copy()
print(adata.shape)
del bdata

# set gene name as var_names 
adata.var['gene_id']=adata.var_names.copy()
adata.var.set_index('gene_name',inplace=True)
adata.var_names=adata.var_names.astype('str')
adata.var_names_make_unique()

cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP','RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2','USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67','TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1','CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF','NEK2','G2E3','GAS2L3','CBX5','CENPA']
s_genes = cell_cycle_genes[:43]
s_genes = [x for x in s_genes if x in adata.var_names]
print('S genes:', s_genes, '\n')
g2m_genes = cell_cycle_genes[43:]
g2m_genes = [x for x in g2m_genes if x in adata.var_names]
print('G2 genes:', g2m_genes, '\n')
cell_cycle_genes = [x for x in cell_cycle_genes if x in adata.var_names]

hb_genes1 = ['HBB','HBD','HBG1','HBG2','HBE1','HBZ','HBM','HBA2','HBA1', 'HBQ1']
hb_genes1 = [x for x in hb_genes1 if x in adata.var_names]
print('Hemoglobin genes set 1:', hb_genes1, '\n')

hb_genes2 = ['AHSP','GYPA','RHAG','GFI1B','KEL','EPB42','ALAS2','TSPO2','GYPB','GYPE','SPTA1','RHCE','KLF1','RHD','SLC4A1','SNX22','SQSTM1','SOD1','FTH1','MAP1LC3B','SLC25A37','TRIM58','HMBS']
hb_genes2 = [x for x in hb_genes2 if x in adata.var_names]
print('Hemoglobin genes set 2:', hb_genes2, '\n')

mito_genes=['MT-ND1','MT-ND2','MT-CO1','MT-CO2','MT-ATP8','MT-ATP6','MT-CO3','MT-ND3','MT-ND4L', 'MT-ND4', 'MT-ND5', 'MT-ND6', 'MT-CYB']
mito_genes = [x for x in mito_genes if x in adata.var_names]
print('Mitochondrial genes:', mito_genes, '\n')

ribo_genes = ['RPLP0','RPLP1','RPLP2','RPL3','RPL3L','RPL4','RPL5','RPL6','RPL7','RPL7A','RPL7L1','RPL8','RPL9','RPL10','RPL10A','RPL10L','RPL11','RPL12','RPL13','RPL13A','RPL14','RPL15','RPL17','RPL18','RPL18A','RPL19','RPL21','RPL22','RPL22L1','RPL23','RPL23A','RPL24','RPL26','RPL26L1','RPL27','RPL27A','RPL28','RPL29','RPL30','RPL31','RPL32','RPL34','RPL35','RPL35A','RPL36','RPL36A','RPL36AL','RPL37','RPL37A','RPL38','RPL39','RPL39L','UBA52','RPL41','RPSA','RPS2','RPS3','RPS3A','RPS4X','RPS4Y1','RPS4Y2','RPS5','RPS6','RPS7','RPS8','RPS9','RPS10','RPS11','RPS12','RPS13','RPS14','RPS15','RPS15A','RPS16','RPS17','RPS18','RPS19','RPS20','RPS21','RPS23','RPS24','RPS25','RPS26','RPS27','RPS27A','RPS27L','RPS28','RPS29','FAU']
ribo_genes = [x for x in ribo_genes if x in adata.var_names]
print('Ribo genes:', ribo_genes, '\n')

stress_genes=['FOS','FOSB','JUN','JUNB','JUND','ZFP36','SOCS3','CEBPD','CEBPB','ATF3','EGR1','HSPA1A','HSPA1B','HSP90AA1','HSP90AB1','HSPA8','HSPB1','HSPE1','IER3','IER2','BTG1','BTG2','DUSP1']
stress_genes = [x for x in stress_genes if x in adata.var_names]
print('Tissue dissociation stress associated genes:', stress_genes, '\n')

# preserve raw count
adata.layers["counts"] = adata.X.copy()

# normalise and log-transform
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5, batch_key='batch_key')

print(sum(adata.var.highly_variable))
sc.pl.highly_variable_genes(adata)

# freeze the state in `.raw`
adata.raw = adata

# subset hvgs
adata = adata[:, adata.var.highly_variable]

adata=adata.copy()

sc.tl.score_genes_cell_cycle(adata, s_genes=s_genes, g2m_genes=g2m_genes)
#sc.pp.regress_out(adata, ['S_score','G2M_score'])

#batch correction
scvi.model.SCVI.setup_anndata(adata, 
                              layer="counts", 
                              batch_key="batch_key",
                              categorical_covariate_keys=['phase'],
                              continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo'])

vae = scvi.model.SCVI(adata, n_hidden = 128, n_latent = 50, n_layers = 3, dispersion = 'gene-batch')
vae.train(max_epochs=50)
    
# Convergence plot
x = np.linspace(1, (len(vae.history['elbo_train'])), (len(vae.history['elbo_train'])))
plt.plot(x, vae.history['elbo_train'], label="train")
plt.xlabel('epoch')
plt.ylabel('elbo_train')
plt.show()
plt.clf()

# add the latent space to adata
adata.obsm["X_scVI"] = vae.get_latent_representation()

#sc.tl.score_genes(adata, stress_genes, score_name='stress_score')

sc.tl.score_genes(adata, hb_genes1, score_name='hb1_score')

sc.pp.neighbors(adata, use_rep="X_scVI")
sc.tl.umap(adata,min_dist=0.1)
#sc.tl.tsne(adata)

sc.pl.umap(adata, color=['log1p_total_counts','n_genes','n_genes_by_counts','total_counts','pct_counts_mt','pct_counts_ribo'],ncols=5)
sc.pl.umap(adata, color=['scrublet_score', 'hb1_score','phase'])
sc.pl.umap(adata, color=['donor','age','region','phase','cell_or_nuclei','modality','kit_10x'],wspace=0.8)
```

Example 41 (python):
```python
Genes=['TTN','TNNT2','MYOM1','LDB3','ALDH1A2','TNNT1','KRT19','UPK3B','BNC1','COL6A3','FBLN1','PCOLCE','LRRC17','PCDH18','EMCN','ECSCR','PECAM1','CDH5','ESAM', 'LAPTM5','PTPRC','LCP1','CD53','WAS','PHOX2B','NRXN1','STMN2','GRIK2','L1CAM']
sc.pl.umap(adata, color=Genes)

#Ito cells
sc.pl.umap(adata, color=['OIT3','CRHBP','FCN3'])

#Parathyroid cells
sc.pl.umap(adata, color=['PTH','MAFB','GATA3'])

#Spleen
sc.pl.umap(adata, color=['VPREB3','MZB1','CD79B'])

#Hepatocytes
sc.pl.umap(adata, color=['ALB','APOA1','TTR'])

#Melanocytes
sc.pl.umap(adata, color=['MLANA','DCT','DCD','ALX1','CCER2'])

#Adipocytes
sc.pl.umap(adata, color=['GPAM','FASN','ADIPOQ'])

#Tenocytes
sc.pl.umap(adata, color=['TNMD','SCX','TNC','CNMD','THBS4'])

#Chondrocyte markers
sc.pl.umap(adata, color=['MATN4','ACAN','CHAD','COL6A3','COL9A1','COL9A3','HAPLN1','CYTL1','COL2A1','COMP','MKX','SOX9','CD74','EFEMP1','TGM2'])

#AT/12 cells
sc.pl.umap(adata, color=['EPCAM','HOPX','LAMP3','SFTPC'])

#Neural crest cells
sc.pl.umap(adata, color=['FOXD3','SNAI1','SNAI2','SOX8','SOX9','SOX10','TFAP2A'])

#Placenta cytotrophoblasts
sc.pl.umap(adata, color=['PAGE4','GSTA3'])

#Esophagus epithelial
sc.pl.umap(adata, color=['SOX2','TP63','TRH'])
```

Example 42 (python):
```python
sc.pl.umap(adata, color=['MYOM1','TNNT2'])
sc.tl.leiden(adata, resolution=0.5, key_added='leiden') #Cluster 14 hepatocytes, cluster 10 and 6 cardios
sc.pl.umap(adata, color='leiden',legend_loc='on data')
```

Example 43 (python):
```python
#this block is to see where the cells over a certain threshold with TNNT2 expression lies in the UMAP
#enter what gene to set threshold for
#when running this box for the first time, comment the starting with 'del'
#then uncomment it for the other ones --- !!!

sc.pl.umap(adata, color=['TNNT2'])
CellName='Cardio_toremove'
gene='TNNT2'
exp_thresh=2
gene_pos=adata.var_names.get_loc(gene)
adata_sel = adata[adata.X[:,gene_pos]>exp_thresh].copy()

l=adata_sel.obs_names
set(l).intersection(adata.obs_names)
print(len(l))

#del adata.obs[CellName]
adata.obs.loc[set(l).intersection(adata.obs_names),CellName]='Yes'
sc.settings.set_figure_params(dpi=200, facecolor='white', color_map = 'RdPu',)
sc.pl.umap(adata, color=[CellName], groups='Yes',size=10, wspace=0.45, legend_fontsize=5, legend_loc='on data',frameon=False)
```

Example 44 (python):
```python
bdata = adata[adata.obs['leiden'].isin(['6','11'])].copy()
bdatanames=bdata.obs_names #this is for the obvious clusters to remove directly
cdatanames=adata_sel.obs_names #this is from the removed cells above

p=set(cdatanames)
r=set(bdatanames)
p.update(r)
p=list(p)

print(len(p))
np.savetxt('./Trisomy/RemovedCells/RBCandCardioRemovedFromNeural.txt', p, delimiter="\n", fmt="%s")
```

Example 45 (python):
```python
adata=sc.read('./Objects/T21_multiome_RNA_foetal.h5ad')
print(adata.shape)

bdata=sc.read('./Objects/T21_multiome_RNA_foetal_scVIProcessed.h5ad')
print(bdata.shape)
#Select Cardiomyocyte, Mesenchymal, Endothelium, Epicardium, Immune, Neural
bdata=bdata[bdata.obs.coarse_grain=='Mesenchymal',:].copy()

adata=adata[[x for x in adata.obs_names if x in bdata.obs_names],:].copy()
print(adata.shape)
del bdata

# set gene name as var_names 
adata.var['gene_id']=adata.var_names.copy()
adata.var.set_index('gene_name',inplace=True)
adata.var_names=adata.var_names.astype('str')
adata.var_names_make_unique()

cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP','RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2','USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67','TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1','CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF','NEK2','G2E3','GAS2L3','CBX5','CENPA']
s_genes = cell_cycle_genes[:43]
s_genes = [x for x in s_genes if x in adata.var_names]
print('S genes:', s_genes, '\n')
g2m_genes = cell_cycle_genes[43:]
g2m_genes = [x for x in g2m_genes if x in adata.var_names]
print('G2 genes:', g2m_genes, '\n')
cell_cycle_genes = [x for x in cell_cycle_genes if x in adata.var_names]

hb_genes1 = ['HBB','HBD','HBG1','HBG2','HBE1','HBZ','HBM','HBA2','HBA1', 'HBQ1']
hb_genes1 = [x for x in hb_genes1 if x in adata.var_names]
print('Hemoglobin genes set 1:', hb_genes1, '\n')

hb_genes2 = ['AHSP','GYPA','RHAG','GFI1B','KEL','EPB42','ALAS2','TSPO2','GYPB','GYPE','SPTA1','RHCE','KLF1','RHD','SLC4A1','SNX22','SQSTM1','SOD1','FTH1','MAP1LC3B','SLC25A37','TRIM58','HMBS']
hb_genes2 = [x for x in hb_genes2 if x in adata.var_names]
print('Hemoglobin genes set 2:', hb_genes2, '\n')

mito_genes=['MT-ND1','MT-ND2','MT-CO1','MT-CO2','MT-ATP8','MT-ATP6','MT-CO3','MT-ND3','MT-ND4L', 'MT-ND4', 'MT-ND5', 'MT-ND6', 'MT-CYB']
mito_genes = [x for x in mito_genes if x in adata.var_names]
print('Mitochondrial genes:', mito_genes, '\n')

ribo_genes = ['RPLP0','RPLP1','RPLP2','RPL3','RPL3L','RPL4','RPL5','RPL6','RPL7','RPL7A','RPL7L1','RPL8','RPL9','RPL10','RPL10A','RPL10L','RPL11','RPL12','RPL13','RPL13A','RPL14','RPL15','RPL17','RPL18','RPL18A','RPL19','RPL21','RPL22','RPL22L1','RPL23','RPL23A','RPL24','RPL26','RPL26L1','RPL27','RPL27A','RPL28','RPL29','RPL30','RPL31','RPL32','RPL34','RPL35','RPL35A','RPL36','RPL36A','RPL36AL','RPL37','RPL37A','RPL38','RPL39','RPL39L','UBA52','RPL41','RPSA','RPS2','RPS3','RPS3A','RPS4X','RPS4Y1','RPS4Y2','RPS5','RPS6','RPS7','RPS8','RPS9','RPS10','RPS11','RPS12','RPS13','RPS14','RPS15','RPS15A','RPS16','RPS17','RPS18','RPS19','RPS20','RPS21','RPS23','RPS24','RPS25','RPS26','RPS27','RPS27A','RPS27L','RPS28','RPS29','FAU']
ribo_genes = [x for x in ribo_genes if x in adata.var_names]
print('Ribo genes:', ribo_genes, '\n')

stress_genes=['FOS','FOSB','JUN','JUNB','JUND','ZFP36','SOCS3','CEBPD','CEBPB','ATF3','EGR1','HSPA1A','HSPA1B','HSP90AA1','HSP90AB1','HSPA8','HSPB1','HSPE1','IER3','IER2','BTG1','BTG2','DUSP1']
stress_genes = [x for x in stress_genes if x in adata.var_names]
print('Tissue dissociation stress associated genes:', stress_genes, '\n')

# preserve raw count
adata.layers["counts"] = adata.X.copy()

# normalise and log-transform
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5, batch_key='batch_key')

print(sum(adata.var.highly_variable))
sc.pl.highly_variable_genes(adata)

# freeze the state in `.raw`
adata.raw = adata

# subset hvgs
adata = adata[:, adata.var.highly_variable]

adata=adata.copy()

sc.tl.score_genes_cell_cycle(adata, s_genes=s_genes, g2m_genes=g2m_genes)
#sc.pp.regress_out(adata, ['S_score','G2M_score'])

#batch correction
scvi.model.SCVI.setup_anndata(adata, 
                              layer="counts", 
                              batch_key="batch_key",
                              categorical_covariate_keys=['phase'],
                              continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo'])

vae = scvi.model.SCVI(adata, n_hidden = 128, n_latent = 50, n_layers = 3, dispersion = 'gene-batch')
vae.train(max_epochs=50)
    
# Convergence plot
x = np.linspace(1, (len(vae.history['elbo_train'])), (len(vae.history['elbo_train'])))
plt.plot(x, vae.history['elbo_train'], label="train")
plt.xlabel('epoch')
plt.ylabel('elbo_train')
plt.show()
plt.clf()

# add the latent space to adata
adata.obsm["X_scVI"] = vae.get_latent_representation()

#sc.tl.score_genes(adata, stress_genes, score_name='stress_score')

sc.tl.score_genes(adata, hb_genes1, score_name='hb1_score')

sc.pp.neighbors(adata, use_rep="X_scVI")
sc.tl.umap(adata,min_dist=0.1)
#sc.tl.tsne(adata)

sc.pl.umap(adata, color=['log1p_total_counts','n_genes','n_genes_by_counts','total_counts','pct_counts_mt','pct_counts_ribo'],ncols=5)
sc.pl.umap(adata, color=['scrublet_score', 'hb1_score','phase'])
sc.pl.umap(adata, color=['donor','age','region','phase','cell_or_nuclei','modality','kit_10x'],wspace=0.8)
```

Example 46 (python):
```python
Genes=['TTN','TNNT2','MYOM1','LDB3','ALDH1A2','TNNT1','KRT19','UPK3B','BNC1','COL6A3','FBLN1','PCOLCE','LRRC17','PCDH18','EMCN','ECSCR','PECAM1','CDH5','ESAM', 'LAPTM5','PTPRC','LCP1','CD53','WAS','PHOX2B','NRXN1','STMN2','GRIK2','L1CAM']
sc.pl.umap(adata, color=Genes)

#Ito cells
sc.pl.umap(adata, color=['OIT3','CRHBP','FCN3'])

#Parathyroid cells
sc.pl.umap(adata, color=['PTH','MAFB','GATA3'])

#Spleen
sc.pl.umap(adata, color=['VPREB3','MZB1','CD79B'])

#Hepatocytes
sc.pl.umap(adata, color=['ALB','APOA1','TTR'])

#Melanocytes
sc.pl.umap(adata, color=['MLANA','DCT','DCD','ALX1','CCER2'])

#Adipocytes
sc.pl.umap(adata, color=['GPAM','FASN','ADIPOQ'])

#Tenocytes
sc.pl.umap(adata, color=['TNMD','SCX','TNC','CNMD','THBS4'])

#Chondrocyte markers
sc.pl.umap(adata, color=['MATN4','ACAN','CHAD','COL6A3','COL9A1','COL9A3','HAPLN1','CYTL1','COL2A1','COMP','MKX','SOX9','CD74','EFEMP1','TGM2'])

#AT/12 cells
sc.pl.umap(adata, color=['EPCAM','HOPX','LAMP3','SFTPC'])

#Neural crest cells
sc.pl.umap(adata, color=['FOXD3','SNAI1','SNAI2','SOX8','SOX9','SOX10','TFAP2A'])

#Placenta cytotrophoblasts
sc.pl.umap(adata, color=['PAGE4','GSTA3'])

#Esophagus epithelial
sc.pl.umap(adata, color=['SOX2','TP63','TRH'])
```

Example 47 (python):
```python
sc.pl.umap(adata, color=['PDGFRA','MYH11','DCN','POSTN','TCF21','KCNJ8','PLA2G5','MT2A','HIGD1B','CLU','PDGFRB'])
```

Example 48 (python):
```python
sc.pl.umap(adata, color=hb_genes1)
sc.pl.umap(adata, color=hb_genes2)

for k in list(set(adata.obs.region)):
    sc.pl.umap(adata, color='region',groups=k)
    
for k in list(set(adata.obs.sangerID)):
    sc.pl.umap(adata, color='sangerID',groups=k)
```

Example 49 (python):
```python
sc.tl.leiden(adata, resolution=0.3, key_added='leiden')
sc.pl.umap(adata, color='leiden',legend_loc='on data')
```

Example 50 (python):
```python
adata = adata[adata.obs['leiden'].isin(['0','1','2','3','4','5','6','7','8','10','11','13'])].copy()
```

Example 51 (python):
```python
#this block is to see where the cells over a certain threshold with TNNT2 expression lies in the UMAP
#enter what gene to set threshold for
#when running this box for the first time, comment the starting with 'del'
#then uncomment it for the other ones --- !!!

sc.pl.umap(adata, color=['TNNT2'])
CellName='Cardio_toremove'
gene='TNNT2'
exp_thresh=2
gene_pos=adata.var_names.get_loc(gene)
adata_sel = adata[adata.X[:,gene_pos]>exp_thresh].copy()

l=adata_sel.obs_names
set(l).intersection(adata.obs_names)
print(len(l))

#del adata.obs[CellName]
adata.obs.loc[set(l).intersection(adata.obs_names),CellName]='Yes'
sc.settings.set_figure_params(dpi=200, facecolor='white', color_map = 'RdPu',)
sc.pl.umap(adata, color=[CellName], groups='Yes',size=10, wspace=0.45, legend_fontsize=5, legend_loc='on data',frameon=False)
```

Example 52 (python):
```python
bdatanames=adata_sel.obs_names #this is from the removed cells above
len(bdatanames)
np.savetxt('./Trisomy/RemovedCells/RBCandCardioRemovedMesenchymal.txt', bdatanames, delimiter="\n", fmt="%s")
```

Example 53 (python):
```python
adata=sc.read('./Objects/T21_multiome_RNA_foetal.h5ad')
print(adata.shape)

bdata=sc.read('./Objects/T21_multiome_RNA_foetal_scVIProcessed.h5ad')
print(bdata.shape)
#Select Cardiomyocyte, Mesenchymal, Endothelium, Epicardium, Immune, Neural
bdata=bdata[bdata.obs.coarse_grain=='Cardiomyocytes',:].copy()

adata=adata[[x for x in adata.obs_names if x in bdata.obs_names],:].copy()
print(adata.shape)
del bdata

# set gene name as var_names 
adata.var['gene_id']=adata.var_names.copy()
adata.var.set_index('gene_name',inplace=True)
adata.var_names=adata.var_names.astype('str')
adata.var_names_make_unique()

cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP','RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2','USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67','TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1','CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF','NEK2','G2E3','GAS2L3','CBX5','CENPA']
s_genes = cell_cycle_genes[:43]
s_genes = [x for x in s_genes if x in adata.var_names]
print('S genes:', s_genes, '\n')
g2m_genes = cell_cycle_genes[43:]
g2m_genes = [x for x in g2m_genes if x in adata.var_names]
print('G2 genes:', g2m_genes, '\n')
cell_cycle_genes = [x for x in cell_cycle_genes if x in adata.var_names]

hb_genes1 = ['HBB','HBD','HBG1','HBG2','HBE1','HBZ','HBM','HBA2','HBA1', 'HBQ1']
hb_genes1 = [x for x in hb_genes1 if x in adata.var_names]
print('Hemoglobin genes set 1:', hb_genes1, '\n')

hb_genes2 = ['AHSP','GYPA','RHAG','GFI1B','KEL','EPB42','ALAS2','TSPO2','GYPB','GYPE','SPTA1','RHCE','KLF1','RHD','SLC4A1','SNX22','SQSTM1','SOD1','FTH1','MAP1LC3B','SLC25A37','TRIM58','HMBS']
hb_genes2 = [x for x in hb_genes2 if x in adata.var_names]
print('Hemoglobin genes set 2:', hb_genes2, '\n')

mito_genes=['MT-ND1','MT-ND2','MT-CO1','MT-CO2','MT-ATP8','MT-ATP6','MT-CO3','MT-ND3','MT-ND4L', 'MT-ND4', 'MT-ND5', 'MT-ND6', 'MT-CYB']
mito_genes = [x for x in mito_genes if x in adata.var_names]
print('Mitochondrial genes:', mito_genes, '\n')

ribo_genes = ['RPLP0','RPLP1','RPLP2','RPL3','RPL3L','RPL4','RPL5','RPL6','RPL7','RPL7A','RPL7L1','RPL8','RPL9','RPL10','RPL10A','RPL10L','RPL11','RPL12','RPL13','RPL13A','RPL14','RPL15','RPL17','RPL18','RPL18A','RPL19','RPL21','RPL22','RPL22L1','RPL23','RPL23A','RPL24','RPL26','RPL26L1','RPL27','RPL27A','RPL28','RPL29','RPL30','RPL31','RPL32','RPL34','RPL35','RPL35A','RPL36','RPL36A','RPL36AL','RPL37','RPL37A','RPL38','RPL39','RPL39L','UBA52','RPL41','RPSA','RPS2','RPS3','RPS3A','RPS4X','RPS4Y1','RPS4Y2','RPS5','RPS6','RPS7','RPS8','RPS9','RPS10','RPS11','RPS12','RPS13','RPS14','RPS15','RPS15A','RPS16','RPS17','RPS18','RPS19','RPS20','RPS21','RPS23','RPS24','RPS25','RPS26','RPS27','RPS27A','RPS27L','RPS28','RPS29','FAU']
ribo_genes = [x for x in ribo_genes if x in adata.var_names]
print('Ribo genes:', ribo_genes, '\n')

stress_genes=['FOS','FOSB','JUN','JUNB','JUND','ZFP36','SOCS3','CEBPD','CEBPB','ATF3','EGR1','HSPA1A','HSPA1B','HSP90AA1','HSP90AB1','HSPA8','HSPB1','HSPE1','IER3','IER2','BTG1','BTG2','DUSP1']
stress_genes = [x for x in stress_genes if x in adata.var_names]
print('Tissue dissociation stress associated genes:', stress_genes, '\n')

# preserve raw count
adata.layers["counts"] = adata.X.copy()

# normalise and log-transform
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5, batch_key='batch_key')

print(sum(adata.var.highly_variable))
sc.pl.highly_variable_genes(adata)

# freeze the state in `.raw`
adata.raw = adata

# subset hvgs
adata = adata[:, adata.var.highly_variable]

adata=adata.copy()

sc.tl.score_genes_cell_cycle(adata, s_genes=s_genes, g2m_genes=g2m_genes)
#sc.pp.regress_out(adata, ['S_score','G2M_score'])

#batch correction
scvi.model.SCVI.setup_anndata(adata, 
                              layer="counts", 
                              batch_key="batch_key",
                              categorical_covariate_keys=['phase'],
                              continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo'])

vae = scvi.model.SCVI(adata, n_hidden = 128, n_latent = 50, n_layers = 3, dispersion = 'gene-batch')
vae.train(max_epochs=50)
    
# Convergence plot
x = np.linspace(1, (len(vae.history['elbo_train'])), (len(vae.history['elbo_train'])))
plt.plot(x, vae.history['elbo_train'], label="train")
plt.xlabel('epoch')
plt.ylabel('elbo_train')
plt.show()
plt.clf()

# add the latent space to adata
adata.obsm["X_scVI"] = vae.get_latent_representation()

#sc.tl.score_genes(adata, stress_genes, score_name='stress_score')

sc.tl.score_genes(adata, hb_genes1, score_name='hb1_score')

sc.pp.neighbors(adata, use_rep="X_scVI")
sc.tl.umap(adata,min_dist=0.1)
#sc.tl.tsne(adata)

sc.pl.umap(adata, color=['log1p_total_counts','n_genes','n_genes_by_counts','total_counts','pct_counts_mt','pct_counts_ribo'],ncols=5)
sc.pl.umap(adata, color=['scrublet_score', 'hb1_score','phase'])
sc.pl.umap(adata, color=['donor','age','region','phase','cell_or_nuclei','modality','kit_10x'],wspace=0.8)
```

Example 54 (python):
```python
for k in list(set(adata.obs.region)):
    sc.pl.umap(adata, color='region',groups=k)
    
for k in list(set(adata.obs.sangerID)):
    sc.pl.umap(adata, color='sangerID',groups=k)
```

Example 55 (python):
```python
Genes=['TTN','TNNT2','MYOM1','LDB3','ALDH1A2','TNNT1','KRT19','UPK3B','BNC1','COL6A3','FBLN1','PCOLCE','LRRC17','PCDH18','EMCN','ECSCR','PECAM1','CDH5','ESAM', 'LAPTM5','PTPRC','LCP1','CD53','WAS','PHOX2B','NRXN1','STMN2','GRIK2','L1CAM']
sc.pl.umap(adata, color=Genes)

#Ito cells
sc.pl.umap(adata, color=['OIT3','CRHBP','FCN3'])

#Parathyroid cells
sc.pl.umap(adata, color=['PTH','MAFB','GATA3'])

#Spleen
sc.pl.umap(adata, color=['VPREB3','MZB1','CD79B'])

#Hepatocytes
sc.pl.umap(adata, color=['ALB','APOA1','TTR'])

#Melanocytes
sc.pl.umap(adata, color=['MLANA','DCT','DCD','ALX1','CCER2'])

#Adipocytes
sc.pl.umap(adata, color=['GPAM','FASN','ADIPOQ'])

#Tenocytes
sc.pl.umap(adata, color=['TNMD','SCX','TNC','CNMD','THBS4'])

#Chondrocyte markers
sc.pl.umap(adata, color=['MATN4','ACAN','CHAD','COL6A3','COL9A1','COL9A3','HAPLN1','CYTL1','COL2A1','COMP','MKX','SOX9','CD74','EFEMP1','TGM2'])

#AT/12 cells
sc.pl.umap(adata, color=['EPCAM','HOPX','LAMP3','SFTPC'])

#Neural crest cells
sc.pl.umap(adata, color=['FOXD3','SNAI1','SNAI2','SOX8','SOX9','SOX10','TFAP2A'])

#Placenta cytotrophoblasts
sc.pl.umap(adata, color=['PAGE4','GSTA3'])

#Esophagus epithelial
sc.pl.umap(adata, color=['SOX2','TP63','TRH'])
```

Example 56 (python):
```python
sc.pl.umap(adata, color=['SHOX2', 'TBX3','BMP10','PITX2'])
```

Example 57 (python):
```python
sc.tl.leiden(adata, resolution=0.22, key_added='leiden')
sc.pl.umap(adata, color='leiden',legend_loc='on data')
```

Example 58 (python):
```python
#Getting the UMAPs for clusterofinterest
ClusterOfInterest='3'
sc.tl.rank_genes_groups(adata, 'leiden', method='t-test', key_added = "t-test")
DEGenesForCluster = [x[ClusterOfInterest] for x in adata.uns['t-test']['names'][:50]]
sc.pl.dotplot(adata, DEGenesForCluster ,groupby='leiden',standard_scale='var',cmap='PRGn')
#sc.pl.umap(adata, color=DEGenesForCluster)
```

Example 59 (python):
```python
sc.tl.rank_genes_groups(adata, 'leiden', groups=['3'], reference='1', method='t-test')
sc.pl.rank_genes_groups(adata, groups=['3'], n_genes=20)
```

Example 60 (python):
```python
adata=sc.read('./Objects/T21_multiome_RNA_foetal.h5ad')
print(adata.shape)

RBCandCardioRemovedFromEndo=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedFromEndo.txt')]
RBCandCardioRemovedFromEpi=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedFromEpi.txt')]
RBCandCardioRemovedFromImmune=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedFromImmune.txt')]
RBCandCardioRemovedFromNeural=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedFromNeural.txt')]
RBCandCardioRemovedMesenchymal=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedMesenchymal.txt')]
ShouldNotBeIn=RBCandCardioRemovedFromEndo+RBCandCardioRemovedFromEpi+RBCandCardioRemovedFromImmune+RBCandCardioRemovedFromNeural+RBCandCardioRemovedMesenchymal
print(len(ShouldNotBeIn))

print(adata)
adata=adata[[x for x in adata.obs_names if x not in ShouldNotBeIn],:].copy()
print(adata)

# set gene name as var_names 
adata.var['gene_id']=adata.var_names.copy()
adata.var.set_index('gene_name',inplace=True)
adata.var_names=adata.var_names.astype('str')
adata.var_names_make_unique()

cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP','RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2','USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67','TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1','CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF','NEK2','G2E3','GAS2L3','CBX5','CENPA']
s_genes = cell_cycle_genes[:43]
s_genes = [x for x in s_genes if x in adata.var_names]
print('S genes:', s_genes, '\n')
g2m_genes = cell_cycle_genes[43:]
g2m_genes = [x for x in g2m_genes if x in adata.var_names]
print('G2 genes:', g2m_genes, '\n')
cell_cycle_genes = [x for x in cell_cycle_genes if x in adata.var_names]

hb_genes1 = ['HBB','HBD','HBG1','HBG2','HBE1','HBZ','HBM','HBA2','HBA1', 'HBQ1']
hb_genes1 = [x for x in hb_genes1 if x in adata.var_names]
print('Hemoglobin genes set 1:', hb_genes1, '\n')

hb_genes2 = ['AHSP','GYPA','RHAG','GFI1B','KEL','EPB42','ALAS2','TSPO2','GYPB','GYPE','SPTA1','RHCE','KLF1','RHD','SLC4A1','SNX22','SQSTM1','SOD1','FTH1','MAP1LC3B','SLC25A37','TRIM58','HMBS']
hb_genes2 = [x for x in hb_genes2 if x in adata.var_names]
print('Hemoglobin genes set 2:', hb_genes2, '\n')

mito_genes=['MT-ND1','MT-ND2','MT-CO1','MT-CO2','MT-ATP8','MT-ATP6','MT-CO3','MT-ND3','MT-ND4L', 'MT-ND4', 'MT-ND5', 'MT-ND6', 'MT-CYB']
mito_genes = [x for x in mito_genes if x in adata.var_names]
print('Mitochondrial genes:', mito_genes, '\n')

ribo_genes = ['RPLP0','RPLP1','RPLP2','RPL3','RPL3L','RPL4','RPL5','RPL6','RPL7','RPL7A','RPL7L1','RPL8','RPL9','RPL10','RPL10A','RPL10L','RPL11','RPL12','RPL13','RPL13A','RPL14','RPL15','RPL17','RPL18','RPL18A','RPL19','RPL21','RPL22','RPL22L1','RPL23','RPL23A','RPL24','RPL26','RPL26L1','RPL27','RPL27A','RPL28','RPL29','RPL30','RPL31','RPL32','RPL34','RPL35','RPL35A','RPL36','RPL36A','RPL36AL','RPL37','RPL37A','RPL38','RPL39','RPL39L','UBA52','RPL41','RPSA','RPS2','RPS3','RPS3A','RPS4X','RPS4Y1','RPS4Y2','RPS5','RPS6','RPS7','RPS8','RPS9','RPS10','RPS11','RPS12','RPS13','RPS14','RPS15','RPS15A','RPS16','RPS17','RPS18','RPS19','RPS20','RPS21','RPS23','RPS24','RPS25','RPS26','RPS27','RPS27A','RPS27L','RPS28','RPS29','FAU']
ribo_genes = [x for x in ribo_genes if x in adata.var_names]
print('Ribo genes:', ribo_genes, '\n')

stress_genes=['FOS','FOSB','JUN','JUNB','JUND','ZFP36','SOCS3','CEBPD','CEBPB','ATF3','EGR1','HSPA1A','HSPA1B','HSP90AA1','HSP90AB1','HSPA8','HSPB1','HSPE1','IER3','IER2','BTG1','BTG2','DUSP1']
stress_genes = [x for x in stress_genes if x in adata.var_names]
print('Tissue dissociation stress associated genes:', stress_genes, '\n')

# preserve raw count
adata.layers["counts"] = adata.X.copy()

# normalise and log-transform
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5, batch_key='batch_key')

print(sum(adata.var.highly_variable))
sc.pl.highly_variable_genes(adata)

# freeze the state in `.raw`
adata.raw = adata

# subset hvgs
adata = adata[:, adata.var.highly_variable]

adata=adata.copy()

sc.tl.score_genes_cell_cycle(adata, s_genes=s_genes, g2m_genes=g2m_genes)
#sc.pp.regress_out(adata, ['S_score','G2M_score'])

#batch correction
scvi.model.SCVI.setup_anndata(adata, 
                              layer="counts", 
                              batch_key="batch_key",
                              categorical_covariate_keys=['phase'],
                              continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo'])

vae = scvi.model.SCVI(adata, n_hidden = 128, n_latent = 50, n_layers = 3, dispersion = 'gene-batch')
vae.train(max_epochs=50)
    
# Convergence plot
x = np.linspace(1, (len(vae.history['elbo_train'])), (len(vae.history['elbo_train'])))
plt.plot(x, vae.history['elbo_train'], label="train")
plt.xlabel('epoch')
plt.ylabel('elbo_train')
plt.show()
plt.clf()

# add the latent space to adata
adata.obsm["X_scVI"] = vae.get_latent_representation()

#sc.tl.score_genes(adata, stress_genes, score_name='stress_score')

sc.tl.score_genes(adata, hb_genes1, score_name='hb1_score')

sc.pp.neighbors(adata, use_rep="X_scVI")
sc.tl.umap(adata,min_dist=0.1)
#sc.tl.tsne(adata)

sc.pl.umap(adata, color=['log1p_total_counts','n_genes','n_genes_by_counts','total_counts','pct_counts_mt','pct_counts_ribo'],ncols=5)
sc.pl.umap(adata, color=['scrublet_score', 'hb1_score','phase'])
sc.pl.umap(adata, color=['donor','age','region','phase','cell_or_nuclei','modality','kit_10x'],wspace=0.8)
```

Example 61 (python):
```python
sc.pl.umap(adata, color=hb_genes1)
sc.pl.umap(adata, color=hb_genes2)

for k in list(set(adata.obs.region)):
    sc.pl.umap(adata, color='region',groups=k)
    
for k in list(set(adata.obs.sangerID)):
    sc.pl.umap(adata, color='sangerID',groups=k)
```

Example 62 (python):
```python
Genes=['TTN','TNNT2','MYOM1','LDB3','ALDH1A2','TNNT1','KRT19','UPK3B','BNC1','COL6A3','FBLN1','PCOLCE','LRRC17','PCDH18','EMCN','ECSCR','PECAM1','CDH5','ESAM', 'LAPTM5','PTPRC','LCP1','CD53','WAS','PHOX2B','NRXN1','STMN2','GRIK2','L1CAM']
sc.pl.umap(adata, color=Genes)

#Ito cells
sc.pl.umap(adata, color=['OIT3','CRHBP','FCN3'])

#Parathyroid cells
sc.pl.umap(adata, color=['PTH','MAFB','GATA3'])

#Spleen
sc.pl.umap(adata, color=['VPREB3','MZB1','CD79B'])

#Hepatocytes
sc.pl.umap(adata, color=['ALB','APOA1','TTR'])

#Melanocytes
sc.pl.umap(adata, color=['MLANA','DCT','DCD','ALX1','CCER2'])

#Adipocytes
sc.pl.umap(adata, color=['GPAM','FASN','ADIPOQ'])

#Tenocytes
sc.pl.umap(adata, color=['TNMD','SCX','TNC','CNMD','THBS4'])

#Chondrocyte markers
sc.pl.umap(adata, color=['MATN4','ACAN','CHAD','COL6A3','COL9A1','COL9A3','HAPLN1','CYTL1','COL2A1','COMP','MKX','SOX9','CD74','EFEMP1','TGM2'])

#AT/12 cells
sc.pl.umap(adata, color=['EPCAM','HOPX','LAMP3','SFTPC'])

#Neural crest cells
sc.pl.umap(adata, color=['FOXD3','SNAI1','SNAI2','SOX8','SOX9','SOX10','TFAP2A'])

#Placenta cytotrophoblasts
sc.pl.umap(adata, color=['PAGE4','GSTA3'])

#Esophagus epithelial
sc.pl.umap(adata, color=['SOX2','TP63','TRH'])
```

Example 63 (python):
```python
sc.tl.leiden(adata, resolution=1, key_added='leiden')
sc.pl.umap(adata, color='leiden',legend_loc='on data')
```

Example 64 (python):
```python
sc.pl.umap(adata, color='leiden',groups='33',legend_loc='on data')
```

Example 65 (python):
```python
ClusterOfInterest='40'
#sc.tl.rank_genes_groups(adata, 'leiden', method='t-test', key_added = "t-test")
DEGenesForCluster = [x[ClusterOfInterest] for x in adata.uns['t-test']['names'][:50]]
sc.pl.dotplot(adata, DEGenesForCluster ,groupby='leiden',standard_scale='var',cmap='PRGn')
#sc.pl.umap(adata, color=DEGenesForCluster)
```

Example 66 (python):
```python
ClusterOfInterest='16' #This is skeletal muscle cluster (17 is obtained because it is from a different run)
sc.tl.rank_genes_groups(adata, 'leiden', method='t-test', key_added = "t-test")
DEGenesForCluster = [x[ClusterOfInterest] for x in adata.uns['t-test']['names'][:50]]
sc.pl.dotplot(adata, DEGenesForCluster ,groupby='leiden',standard_scale='var',cmap='PRGn')
#sc.pl.umap(adata, color=DEGenesForCluster)
```

Example 67 (python):
```python
bdata = adata[adata.obs['leiden'].isin(['33','40'])].copy() #33 other cardio in the mesenchymal cluster; 40 is unknown (tried all the known markers)
bdatanames=bdata.obs_names
print(len(bdatanames))
np.savetxt('./Trisomy/RemovedCells/RBCandCardioRemovedGeneral.txt', bdatanames, delimiter="\n", fmt="%s")
```

Example 68 (python):
```python
adata=sc.read('./Objects/T21_multiome_RNA_foetal.h5ad')
print(adata.shape)

RBCandCardioRemovedFromEndo=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedFromEndo.txt')]
RBCandCardioRemovedFromEpi=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedFromEpi.txt')]
RBCandCardioRemovedFromImmune=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedFromImmune.txt')]
RBCandCardioRemovedFromNeural=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedFromNeural.txt')]
RBCandCardioRemovedMesenchymal=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedMesenchymal.txt')]
RBCandCardioRemovedGeneral=[x.strip() for x in open('./Trisomy/RemovedCells/RBCandCardioRemovedGeneral.txt')]
SkeletalMuscle=[x.strip() for x in open('./Trisomy/RemovedCells/SkeletalMuscle.txt')]
ShouldNotBeIn=RBCandCardioRemovedFromEndo+RBCandCardioRemovedFromEpi+RBCandCardioRemovedFromImmune+RBCandCardioRemovedFromNeural+RBCandCardioRemovedMesenchymal+RBCandCardioRemovedGeneral+SkeletalMuscle
print(len(ShouldNotBeIn))

print(adata)
adata=adata[[x for x in adata.obs_names if x not in ShouldNotBeIn],:].copy()
print(adata)

# set gene name as var_names 
adata.var['gene_id']=adata.var_names.copy()
adata.var.set_index('gene_name',inplace=True)
adata.var_names=adata.var_names.astype('str')
adata.var_names_make_unique()

cell_cycle_genes = ['MCM5','PCNA','TYMS','FEN1','MCM2','MCM4','RRM1','UNG','GINS2','MCM6','CDCA7','DTL','PRIM1','UHRF1','MLF1IP','HELLS','RFC2','RPA2','NASP','RAD51AP1','GMNN','WDR76','SLBP','CCNE2','UBR7','POLD3','MSH2','ATAD2','RAD51','RRM2','CDC45','CDC6','EXO1','TIPIN','DSCC1','BLM','CASP8AP2','USP1','CLSPN','POLA1','CHAF1B','BRIP1','E2F8','HMGB2','CDK1','NUSAP1','UBE2C','BIRC5','TPX2','TOP2A','NDC80','CKS2','NUF2','CKS1B','MKI67','TMPO','CENPF','TACC3','FAM64A','SMC4','CCNB2','CKAP2L','CKAP2','AURKB','BUB1','KIF11','ANP32E','TUBB4B','GTSE1','KIF20B','HJURP','CDCA3','HN1','CDC20','TTK','CDC25C','KIF2C','RANGAP1','NCAPD2','DLGAP5','CDCA2','CDCA8','ECT2','KIF23','HMMR','AURKA','PSRC1','ANLN','LBR','CKAP5','CENPE','CTCF','NEK2','G2E3','GAS2L3','CBX5','CENPA']
s_genes = cell_cycle_genes[:43]
s_genes = [x for x in s_genes if x in adata.var_names]
print('S genes:', s_genes, '\n')
g2m_genes = cell_cycle_genes[43:]
g2m_genes = [x for x in g2m_genes if x in adata.var_names]
print('G2 genes:', g2m_genes, '\n')
cell_cycle_genes = [x for x in cell_cycle_genes if x in adata.var_names]

hb_genes1 = ['HBB','HBD','HBG1','HBG2','HBE1','HBZ','HBM','HBA2','HBA1', 'HBQ1']
hb_genes1 = [x for x in hb_genes1 if x in adata.var_names]
print('Hemoglobin genes set 1:', hb_genes1, '\n')

hb_genes2 = ['AHSP','GYPA','RHAG','GFI1B','KEL','EPB42','ALAS2','TSPO2','GYPB','GYPE','SPTA1','RHCE','KLF1','RHD','SLC4A1','SNX22','SQSTM1','SOD1','FTH1','MAP1LC3B','SLC25A37','TRIM58','HMBS']
hb_genes2 = [x for x in hb_genes2 if x in adata.var_names]
print('Hemoglobin genes set 2:', hb_genes2, '\n')

mito_genes=['MT-ND1','MT-ND2','MT-CO1','MT-CO2','MT-ATP8','MT-ATP6','MT-CO3','MT-ND3','MT-ND4L', 'MT-ND4', 'MT-ND5', 'MT-ND6', 'MT-CYB']
mito_genes = [x for x in mito_genes if x in adata.var_names]
print('Mitochondrial genes:', mito_genes, '\n')

ribo_genes = ['RPLP0','RPLP1','RPLP2','RPL3','RPL3L','RPL4','RPL5','RPL6','RPL7','RPL7A','RPL7L1','RPL8','RPL9','RPL10','RPL10A','RPL10L','RPL11','RPL12','RPL13','RPL13A','RPL14','RPL15','RPL17','RPL18','RPL18A','RPL19','RPL21','RPL22','RPL22L1','RPL23','RPL23A','RPL24','RPL26','RPL26L1','RPL27','RPL27A','RPL28','RPL29','RPL30','RPL31','RPL32','RPL34','RPL35','RPL35A','RPL36','RPL36A','RPL36AL','RPL37','RPL37A','RPL38','RPL39','RPL39L','UBA52','RPL41','RPSA','RPS2','RPS3','RPS3A','RPS4X','RPS4Y1','RPS4Y2','RPS5','RPS6','RPS7','RPS8','RPS9','RPS10','RPS11','RPS12','RPS13','RPS14','RPS15','RPS15A','RPS16','RPS17','RPS18','RPS19','RPS20','RPS21','RPS23','RPS24','RPS25','RPS26','RPS27','RPS27A','RPS27L','RPS28','RPS29','FAU']
ribo_genes = [x for x in ribo_genes if x in adata.var_names]
print('Ribo genes:', ribo_genes, '\n')

stress_genes=['FOS','FOSB','JUN','JUNB','JUND','ZFP36','SOCS3','CEBPD','CEBPB','ATF3','EGR1','HSPA1A','HSPA1B','HSP90AA1','HSP90AB1','HSPA8','HSPB1','HSPE1','IER3','IER2','BTG1','BTG2','DUSP1']
stress_genes = [x for x in stress_genes if x in adata.var_names]
print('Tissue dissociation stress associated genes:', stress_genes, '\n')

# preserve raw count
adata.layers["counts"] = adata.X.copy()

# normalise and log-transform
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3, min_disp=0.5, batch_key='batch_key')

print(sum(adata.var.highly_variable))
sc.pl.highly_variable_genes(adata)

# freeze the state in `.raw`
adata.raw = adata

# subset hvgs
adata = adata[:, adata.var.highly_variable]

adata=adata.copy()

sc.tl.score_genes_cell_cycle(adata, s_genes=s_genes, g2m_genes=g2m_genes)
#sc.pp.regress_out(adata, ['S_score','G2M_score'])

#batch correction
scvi.model.SCVI.setup_anndata(adata, 
                              layer="counts", 
                              batch_key="batch_key",
                              categorical_covariate_keys=['phase'],
                              continuous_covariate_keys=['total_counts','pct_counts_mt','pct_counts_ribo'])

vae = scvi.model.SCVI(adata, n_hidden = 128, n_latent = 50, n_layers = 3, dispersion = 'gene-batch')
vae.train(max_epochs=50)
    
# Convergence plot
x = np.linspace(1, (len(vae.history['elbo_train'])), (len(vae.history['elbo_train'])))
plt.plot(x, vae.history['elbo_train'], label="train")
plt.xlabel('epoch')
plt.ylabel('elbo_train')
plt.show()
plt.clf()

# add the latent space to adata
adata.obsm["X_scVI"] = vae.get_latent_representation()

#sc.tl.score_genes(adata, stress_genes, score_name='stress_score')

sc.tl.score_genes(adata, hb_genes1, score_name='hb1_score')

sc.pp.neighbors(adata, use_rep="X_scVI")
sc.tl.umap(adata,min_dist=0.1)
#sc.tl.tsne(adata)

sc.pl.umap(adata, color=['log1p_total_counts','n_genes','n_genes_by_counts','total_counts','pct_counts_mt','pct_counts_ribo'],ncols=5)
sc.pl.umap(adata, color=['scrublet_score', 'hb1_score','phase'])
sc.pl.umap(adata, color=['donor','age','region','phase','cell_or_nuclei','modality','kit_10x'],wspace=0.8)
```

Example 69 (python):
```python
Genes=['TTN','TNNT2','MYOM1','LDB3','ALDH1A2','TNNT1','KRT19','UPK3B','BNC1','COL6A3','FBLN1','PCOLCE','LRRC17','PCDH18','EMCN','ECSCR','PECAM1','CDH5','ESAM', 'LAPTM5','PTPRC','LCP1','CD53','WAS','PHOX2B','NRXN1','STMN2','GRIK2','L1CAM']
sc.pl.umap(adata, color=Genes)
```

Example 70 (python):
```python
sc.pl.umap(adata, color=['TTN','BNC1','COL6A3','PECAM1','PTPRC','LYVE1','NRXN1'])
```

Example 71 (python):
```python
sc.tl.leiden(adata, resolution=1, key_added='leiden')
sc.pl.umap(adata, color='leiden',legend_loc='on data')
```

Example 72 (python):
```python
#add coarse_grain
CardiomyocyteClusters=['0','5','28','22','32','15','19','1']
MesenchymalClusters=['23','3','10','2','14','7','12','30','27','4','20','8','16','21','9','25']
EndotheliumClusters=['34','11','13','24']
EpicardiumClusters=['17']
ImmuneClusters=['33','31','6','18']
NeuralClusters=['29','26']
adata.obs['coarse_grain']= 'na'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in CardiomyocyteClusters]]='Cardiomyocytes'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in MesenchymalClusters]]='Mesenchymal'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in EndotheliumClusters]]='Endothelium'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in EpicardiumClusters]]='Epicardium'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in ImmuneClusters]]='Immune'
adata.obs['coarse_grain'][[x for x in adata.obs_names if adata.obs.leiden[x] in NeuralClusters]]='Neural'
sc.pl.umap(adata, color='coarse_grain')
```

Example 73 (python):
```python
set(adata.obs.donor)
```

Example 74 (python):
```python
adata.obs['cell_or_nuclei']= 'na'

adata.obs['cell_or_nuclei'][adata.obs['donor']=='Hst42']='nuclei'
adata.obs['cell_or_nuclei'][adata.obs['donor']=='Hst44']='nuclei'

sc.pl.umap(adata, color='cell_or_nuclei')
```

Example 75 (python):
```python
adata.obs['modality']= 'na'

adata.obs['modality'][adata.obs['cell_or_nuclei']=='nuclei']='scMultiome'

sc.pl.umap(adata, color='modality')
```

Example 76 (python):
```python
adata.obs['sex']= 'na'

adata.obs['sex'][adata.obs['donor']=='Hst42']='male'
adata.obs['sex'][adata.obs['donor']=='Hst44']='male'

sc.pl.umap(adata, color='sex')
```

Example 77 (python):
```python
adata.obs['FACSgate']= 'na'
sc.pl.umap(adata, color='FACSgate')
```

Example 78 (python):
```python
#all age taken in PCW and not GA
adata.obs['age']= 'na'

adata.obs['age'][adata.obs['donor']=='Hst42']='14W0D'
adata.obs['age'][adata.obs['donor']=='Hst44']='13W0D'

sc.pl.umap(adata, color='age')
```

Example 79 (python):
```python
#all age taken in PCW and not GA
adata.obs['week']= 'na'

adata.obs['week'][adata.obs['donor']=='Hst42']='14W'
adata.obs['week'][adata.obs['donor']=='Hst44']='13W'

sc.pl.umap(adata, color='week')
```

Example 80 (python):
```python
#all age taken in PCW and not GA
adata.obs['trimester']= 'na'

adata.obs['trimester'][adata.obs['week']=='14W']='secondtrimester'
adata.obs['trimester'][adata.obs['week']=='13W']='secondtrimester'

sc.pl.umap(adata, color='trimester')
```

Example 81 (python):
```python
set(adata.obs.region)
```

Example 82 (python):
```python
adata.obs.replace({'region':{'Great vessels':'great vessels'}}, inplace=True)
adata.obs.replace({'region':{'Heart':'whole sample'}}, inplace=True)
adata.obs.replace({'region':{'Left atria':'left atria'}}, inplace=True)
adata.obs.replace({'region':{'Left ventricle':'left ventricle'}}, inplace=True)
adata.obs.replace({'region':{'Pericardium':'pericardium'}}, inplace=True)
adata.obs.replace({'region':{'Right ventricle':'right ventricle'}}, inplace=True)

sc.pl.umap(adata, color='region')
```

Example 83 (python):
```python
sc.pl.umap(adata,color=['phase'])
adata.obs['cycling']= 'no'
adata.obs['cycling'][[x for x in adata.obs_names if adata.obs.phase[x] in ['G2M','S']]]='yes'
sc.pl.umap(adata,color=['cycling'])
```

Example 84 (python):
```python
del(adata.obs['donor_type'])
del(adata.obs['region_finest'])
del(adata.obs['gender'])
del(adata.obs['flushed'])
del(adata.obs['scrublet_leiden'])
del(adata.obs['cluster_scrublet_score'])
del(adata.obs['_scvi_labels'])
```

Example 85 (python):
```python
sc.pl.umap(adata, color=['donor','age','week','trimester','region','phase','cell_or_nuclei','cycling','modality','kit_10x','FACSgate','coarse_grain'],wspace=0.8,ncols=5)
```

Example 86 (python):
```python
adata.write('./Objects/T21HeartsExtracardiacRemovedIntegratedProcessed.h5ad')
```

Example 87 (python):
```python
adata_raw=sc.read('./Objects/T21_multiome_RNA_foetal.h5ad')
print(adata_raw)

# constructing new anndata with maintaining .obs, .obsm, .obsp, .uns
# overwriting adata_raw
adata_raw=anndata.AnnData(X=adata_raw[adata.obs_names,:].X,
                      obs=adata.obs, # from annotated or subsetted data
                      var=adata_raw.var,
                      obsm=adata.obsm, # from annotated or subsetted data
                      obsp=adata.obsp, # from annotated or subsetted data
                    uns=adata.uns # from annotated or subsetted data
                               )

# set gene name as var_names 
adata_raw.var['gene_id']=adata_raw.var_names.copy()
adata_raw.var.set_index('gene_name',inplace=True)
adata_raw.var_names=adata_raw.var_names.astype('str')
adata_raw.var_names_make_unique()
sc.pl.umap(adata_raw, color=['TNNT2'])

adata=adata_raw.copy()
print(adata)
adata.write('./Objects/T21HeartsExtracardiacRemovedRaw.h5ad')
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/QC/Foetal_T21-Multiome-RNA_No1_make-anndata_run-scrublet.html

**Contents:**
- Import modules¶
- Functions¶
- Create anndata object¶
- Run scrublet¶
- Save¶

190750 rows × 21 columns

**Examples:**

Example 1 (python):
```python
import numpy as np
import pandas as pd
import scanpy as sc
import matplotlib.pyplot as plt
import scrublet as scr
import session_info
```

Example 2 (python):
```python
sc.settings.verbosity = 3
sc.settings.set_figure_params(dpi = 160, color_map = 'RdPu', dpi_save = 180, vector_friendly = True, format = 'svg')
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.3.4
numpy               1.20.1
pandas              1.1.4
scanpy              1.7.1
scrublet            NA
session_info        1.0.0
-----
```

Example 4 (unknown):
```unknown
PIL                 8.1.2
anndata             0.7.5
anyio               NA
attr                20.3.0
babel               2.9.0
backcall            0.2.0
brotli              NA
cairo               1.20.0
certifi             2020.12.05
cffi                1.14.5
chardet             4.0.0
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.0
dask                2021.03.1
dateutil            2.8.1
decorator           4.4.2
fsspec              0.8.7
get_version         2.1
google              NA
h5py                2.10.0
idna                2.10
igraph              0.8.3
ipykernel           5.5.0
ipython_genutils    0.2.0
jedi                0.18.0
jinja2              2.11.3
joblib              1.0.1
json5               NA
jsonschema          3.2.0
jupyter_server      1.4.1
jupyterlab_server   2.3.0
kiwisolver          1.3.1
legacy_api_wrap     0.0.0
leidenalg           0.8.3
llvmlite            0.34.0
louvain             0.7.0
markupsafe          1.1.1
mpl_toolkits        NA
natsort             7.1.1
nbclassic           NA
nbformat            5.1.2
numba               0.51.2
numexpr             2.7.3
packaging           20.9
parso               0.8.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
prometheus_client   NA
prompt_toolkit      3.0.16
psutil              5.8.0
ptyprocess          0.7.0
pvectorc            NA
pyarrow             0.16.0
pygments            2.8.0
pyparsing           2.4.7
pyrsistent          NA
pytoml              NA
pytz                2021.1
requests            2.25.1
ruamel              NA
scipy               1.6.1
send2trash          NA
setuptools_scm      NA
simplejson          3.17.2
sinfo               0.3.1
six                 1.15.0
sklearn             0.22
sniffio             1.2.0
socks               1.7.1
sphinxcontrib       NA
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
tlz                 0.11.0
toolz               0.11.1
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
uritemplate         3.0.1
urllib3             1.26.3
wcwidth             0.2.5
yaml                5.1.2
zmq                 22.0.3
```

Example 5 (unknown):
```unknown
-----
IPython             7.21.0
jupyter_client      6.1.11
jupyter_core        4.7.1
jupyterlab          3.0.9
notebook            6.2.0
-----
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0]
Linux-4.15.0-112-generic-x86_64-with-glibc2.10
-----
Session information updated at 2023-03-14 16:45
```

Example 6 (python):
```python
"""Functions for downstream work with outputs of remove-background."""

# to read cellbender outputs<br>
# https://github.com/broadinstitute/CellBender/issues/57

import tables
import numpy as np
import scipy.sparse as sp
from typing import Dict


def dict_from_h5(file: str) -> Dict[str, np.ndarray]:
    """Read in everything from an h5 file and put into a dictionary."""
    d = {}
    with tables.open_file(file) as f:
        # read in everything
        for array in f.walk_nodes("/", "Array"):
            d[array.name] = array.read()
    return d


def anndata_from_h5(file: str,
                    analyzed_barcodes_only: bool = True) -> 'anndata.AnnData':
    """Load an output h5 file into an AnnData object for downstream work.

    Args:
        file: The h5 file
        analyzed_barcodes_only: False to load all barcodes, so that the size of
            the AnnData object will match the size of the input raw count matrix.
            True to load a limited set of barcodes: only those analyzed by the
            algorithm. This allows relevant latent variables to be loaded
            properly into adata.obs and adata.obsm, rather than adata.uns.

    Returns:
        adata: The anndata object, populated with inferred latent variables
            and metadata.

    """

    try:
        import anndata
    except ImportError:
        raise ImportError('The anndata package must be installed to use the '
                          'function anndata_from_h5()')

    d = dict_from_h5(file)
    X = sp.csc_matrix((d.pop('data'), d.pop('indices'), d.pop('indptr')),
                      shape=d.pop('shape')).transpose().tocsr()

    if analyzed_barcodes_only:
        if 'barcodes_analyzed_inds' in d.keys():
            X = X[d['barcodes_analyzed_inds'], :]
            d['barcodes'] = d['barcodes'][d['barcodes_analyzed_inds']]
        elif 'barcode_indices_for_latents' in d.keys():
            X = X[d['barcode_indices_for_latents'], :]
            d['barcodes'] = d['barcodes'][d['barcode_indices_for_latents']]
        else:
            print('Warning: analyzed_barcodes_only=True, but the key '
                  '"barcodes_analyzed_inds" or "barcode_indices_for_latents" '
                  'is missing from the h5 file. '
                  'Will output all barcodes, and proceed as if '
                  'analyzed_barcodes_only=False')

    # Construct the count matrix.
    adata = anndata.AnnData(X=X,
                            obs={'barcode': d.pop('barcodes').astype(str)},
                            var={'gene_name': (d.pop('gene_names') if 'gene_names' in d.keys()
                                               else d.pop('name')).astype(str)})
    adata.obs.set_index('barcode', inplace=True)
    adata.var.set_index('gene_name', inplace=True)

    # Add other information to the adata object in the appropriate slot.
    for key, value in d.items():
        try:
            value = np.asarray(value)
            if len(value.shape) == 0:
                adata.uns[key] = value
            elif value.shape[0] == X.shape[0]:
                if (len(value.shape) < 2) or (value.shape[1] < 2):
                    adata.obs[key] = value
                else:
                    adata.obsm[key] = value
            elif value.shape[0] == X.shape[1]:
                if value.dtype.name.startswith('bytes'):
                    adata.var[key] = value.astype(str)
                else:
                    adata.var[key] = value
            else:
                adata.uns[key] = value
        except Exception:
            print('Unable to load data into AnnData: ', key, value, type(value))

    if analyzed_barcodes_only:
        for col in adata.obs.columns[adata.obs.columns.str.startswith('barcodes_analyzed')
                                     | adata.obs.columns.str.startswith('barcode_indices')]:
            try:
                del adata.obs[col]
            except Exception:
                pass

    return adata
```

Example 7 (python):
```python
# from https://github.com/Teichlab/mapcloud/blob/master/scripts/starsolo/postprocess.py

from statsmodels.stats.multitest import multipletests
# from emptydrops.matrix import CountMatrix
# from emptydrops import find_nonambient_barcodes
import scrublet as scr
import scanpy as sc
import pandas as pd
import numpy as np
import scipy

#some functions that Ni uses in scanpy scripts to run scrublet
#which in turn are inspired by my original notebook on the matter
#(extracted from scanpy_scripts 0.2.10 to get around scanpy version incompatibility)
def test_outlier(x, upper_mad_only=True):
	med = np.median(x)
	if upper_mad_only:
		mad = np.median(x[x>med] - med) * 1.4826
	else:
		mad = np.median(np.abs(x - med)) * 1.4826
	pvals = 1 - scipy.stats.norm.cdf(x, loc=med, scale=mad)
	bh_pvals = multipletests(pvals, method='fdr_bh')[1]
	return pvals, bh_pvals

def run_scrublet(adata, resolution_function=None):
	old_verbosity = sc.settings.verbosity
	sc.settings.verbosity = 1
	if resolution_function is None:
		resolution_function = lambda x: np.maximum(np.maximum(np.log10(x)-1, 0)**2, 0.1)
	scrub = scr.Scrublet(adata.X)
	#this has the potential to brick for poor quality data
	#if so, abort it and everything downstream
	try:
		ds, pd = scrub.scrub_doublets(verbose=False)
	except:
		return
	adata.obs['scrublet_score'] = ds

	adata_copy = adata.copy()
	sc.pp.filter_genes(adata_copy, min_cells=3)
	sc.pp.normalize_total(adata_copy, target_sum=1e4)
	sc.pp.log1p(adata_copy)
	sc.pp.highly_variable_genes(adata_copy, min_mean=0.0125, max_mean=3, min_disp=0.5, subset=True)
	sc.pp.scale(adata_copy, zero_center=False)
	sc.pp.pca(adata_copy, svd_solver='arpack', zero_center=False)
	sc.pp.neighbors(adata_copy, n_pcs=30)
	sc.tl.umap(adata_copy)
	sc.tl.leiden(adata_copy, resolution=1)
	for clst in np.unique(adata_copy.obs['leiden']):
		clst_size = sum(adata_copy.obs['leiden'] == clst)
		sc.tl.leiden(adata_copy, restrict_to=('leiden', [clst]), resolution=resolution_function(clst_size), key_added='leiden_R')
		adata_copy.obs['leiden'] = adata_copy.obs['leiden_R']
	clst_meds = []
	for clst in np.unique(adata_copy.obs['leiden']):
		k = adata_copy.obs['leiden'] == clst
		clst_med = np.median(adata_copy.obs.loc[k, 'scrublet_score'])
		adata_copy.obs.loc[k, 'cluster_scrublet_score'] = clst_med
		clst_meds.append(clst_med)
	clst_meds = np.array(clst_meds)
	pvals, bh_pvals = test_outlier(clst_meds)
	for i, clst in enumerate(np.unique(adata_copy.obs['leiden'])):
		k = adata_copy.obs['leiden'] == clst
		adata_copy.obs.loc[k, 'pval'] = pvals[i]
		adata_copy.obs.loc[k, 'bh_pval'] = bh_pvals[i]
	sc.settings.verbosity = old_verbosity
	#need to also export the clustering, for soupx purposes
	adata.obs['scrublet_leiden'] = adata_copy.obs['leiden']
	adata.obs['scrublet_score'] = adata_copy.obs['scrublet_score']
	adata.obs['cluster_scrublet_score'] = adata_copy.obs['cluster_scrublet_score']
	adata.obs['doublet_pval'] = adata_copy.obs['pval']
	adata.obs['doublet_bh_pval'] = adata_copy.obs['bh_pval']
	del adata_copy
```

Example 8 (python):
```python
# read in metadata
metadata = pd.read_csv('/nfs/team205/heart/anndata_objects/Foetal/metadata/HeartTeamSamples_Mappeddata_20230314.csv', sep = ',', index_col = None)
metadata = metadata[metadata['Publication']=='Foetal']

# select healthy data
metadata = metadata[metadata['status']=='Trisomy21']


print(metadata['modality'].value_counts())
```

Example 9 (unknown):
```unknown
Multiome-ATAC    8
Multiome-RNA     8
Visium           4
Name: modality, dtype: int64
```

Example 10 (python):
```python
# select modality
metadata = metadata[metadata['modality']=='Multiome-RNA']
metadata[['sangerID','combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age',
                'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x','flushed']]
```

Example 11 (python):
```python
########### editted: put id to var_names ###########

# read in
adatas=[]
for i in range(len(metadata)):
    print(metadata.iloc[i]['combinedID'])
    
    path=metadata.iloc[i]['CellBender_out']
    adatas.append(anndata_from_h5(path + '/' + path.split('/')[-1] + '_cellbender_out_filtered.h5', analyzed_barcodes_only=False))
    del path
    
    # replace var_names with ensembleID
    adatas[i].var.reset_index(inplace=True)
    if 'id' in adatas[i].var.columns:
        adatas[i].var.set_index('id',inplace=True)
    else:
        adatas[i].var.set_index('genes',inplace=True)
    
    # modify barcodes
    adatas[i].obs.index = metadata.iloc[i]['combinedID'] + '_' + adatas[i].obs.index 
    
    # add metadata
    for col in ['sangerID','combinedID', 'donor', 'donor_type', 'region', 'region_finest', 'age',
                'gender', 'facility', 'cell_or_nuclei', 'modality', 'kit_10x','flushed']:
        adatas[i].obs[col] = metadata.iloc[i][col]


# concatenate
adata = adatas[0].concatenate(adatas[1:], index_unique = None, batch_key=None)
adata.shape
```

Example 12 (unknown):
```unknown
BHF_F_Hea13188319_BHF_F_Hea13187621
BHF_F_Hea13188320_BHF_F_Hea13187622
BHF_F_Hea13188321_BHF_F_Hea13187623
BHF_F_Hea13188322_BHF_F_Hea13187624
BHF_F_Hea13188323_BHF_F_Hea13187625
BHF_F_Hea13188325_BHF_F_Hea13187627
BHF_F_Hea13188326_BHF_F_Hea13187628
BHF_F_Hea13188327_BHF_F_Hea13187629
```

Example 13 (unknown):
```unknown
(190750, 36601)
```

Example 14 (python):
```python
for i in range(len(adatas)):
    print(adatas[i].var.columns)
```

Example 15 (unknown):
```unknown
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
Index(['gene_name', 'ambient_expression', 'feature_type'], dtype='object')
```

Example 16 (python):
```python
adata.obs['combinedID'].value_counts()
```

Example 17 (unknown):
```unknown
BHF_F_Hea13188319_BHF_F_Hea13187621    50000
BHF_F_Hea13188321_BHF_F_Hea13187623    38944
BHF_F_Hea13188325_BHF_F_Hea13187627    23862
BHF_F_Hea13188322_BHF_F_Hea13187624    17452
BHF_F_Hea13188323_BHF_F_Hea13187625    17143
BHF_F_Hea13188327_BHF_F_Hea13187629    16377
BHF_F_Hea13188320_BHF_F_Hea13187622    14000
BHF_F_Hea13188326_BHF_F_Hea13187628    12972
Name: combinedID, dtype: int64
```

Example 18 (python):
```python
%%time

# per rxn
for i,ID in enumerate(adata.obs['combinedID'].unique()):
    print(ID)
    
    ad = adata[adata.obs['combinedID'] == ID].copy()
    run_scrublet(ad)
    if i==0:
        meta = ad.obs
    else:
        meta = pd.concat([meta, ad.obs])
    del ad
```

Example 19 (unknown):
```unknown
BHF_F_Hea13188319_BHF_F_Hea13187621
BHF_F_Hea13188320_BHF_F_Hea13187622
```

Example 20 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 21 (unknown):
```unknown
BHF_F_Hea13188321_BHF_F_Hea13187623
```

Example 22 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 23 (unknown):
```unknown
BHF_F_Hea13188322_BHF_F_Hea13187624
```

Example 24 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 25 (unknown):
```unknown
BHF_F_Hea13188323_BHF_F_Hea13187625
```

Example 26 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 27 (unknown):
```unknown
BHF_F_Hea13188325_BHF_F_Hea13187627
```

Example 28 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 29 (unknown):
```unknown
BHF_F_Hea13188326_BHF_F_Hea13187628
```

Example 30 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 31 (unknown):
```unknown
BHF_F_Hea13188327_BHF_F_Hea13187629
```

Example 32 (unknown):
```unknown
/opt/conda/envs/teichlab/lib/python3.8/site-packages/scrublet/helper_functions.py:321: RuntimeWarning: divide by zero encountered in true_divide
  w.setdiag(float(target_total) / tots_use)
WARNING: Some cells have total count of genes equal to zero
```

Example 33 (unknown):
```unknown
CPU times: user 38min 57s, sys: 18min 16s, total: 57min 13s
Wall time: 17min 13s
```

Example 34 (python):
```python
meta_scrub = meta[['scrublet_score', 'scrublet_leiden', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval']].copy()
meta_scrub.shape
```

Example 35 (unknown):
```unknown
(190750, 5)
```

Example 36 (python):
```python
if meta_scrub.reindex(adata.obs.index).index.equals(adata.obs.index):
    adata.obs = pd.concat([adata.obs, meta_scrub.reindex(adata.obs.index)], axis=1)
else:
    raise Exception('Different barcodes in meta and adata')
```

Example 37 (python):
```python
adata.write('/nfs/team205/heart/anndata_objects/Foetal/QC/T21_multiome_RNA_foetal_prefilter.h5ad')
```

Example 38 (unknown):
```unknown
... storing 'sangerID' as categorical
... storing 'combinedID' as categorical
... storing 'donor' as categorical
... storing 'donor_type' as categorical
... storing 'region' as categorical
... storing 'region_finest' as categorical
... storing 'age' as categorical
... storing 'gender' as categorical
... storing 'facility' as categorical
... storing 'cell_or_nuclei' as categorical
... storing 'modality' as categorical
... storing 'kit_10x' as categorical
... storing 'flushed' as categorical
... storing 'scrublet_leiden' as categorical
... storing 'gene_name' as categorical
... storing 'feature_type' as categorical
```

Example 39 (python):
```python
sc.pl.violin(adata,
             ['scrublet_score', 'cluster_scrublet_score', 'doublet_pval', 'doublet_bh_pval'],jitter = 0, multi_panel = True)
```

Example 40 (python):
```python
adata.obs['donor'].value_counts()
```

Example 41 (unknown):
```unknown
Hst42    137539
Hst44     53211
Name: donor, dtype: int64
```

---

## Notebook

**URL:** http://127.0.0.1:9020/notebooks_html/Fig1_General_processing/RNA/QC/Demultiplex/No4_Assign-donors.html

**Contents:**
- Multiplexed samples ID¶
- Read in SOC result¶
- Assign donors with CMO result (for SOC unassigned nuclei)¶
- Make assignment dataframe and save¶

**Examples:**

Example 1 (python):
```python
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

import numpy as np
import pandas as pd
import scanpy as sc
import session_info
import matplotlib.pyplot as plt
import seaborn as sns

import os
```

Example 2 (unknown):
```unknown
session_info.show()
```

Example 3 (python):
```python
-----
matplotlib          3.5.2
numpy               1.21.5
pandas              1.3.5
scanpy              1.8.2
seaborn             0.11.1
session_info        1.0.0
-----
```

Example 4 (python):
```python
PIL                 9.1.1
anndata             0.8.0
asciitree           NA
backcall            0.2.0
beta_ufunc          NA
binom_ufunc         NA
cached_property     1.5.1
cairo               1.20.0
cffi                1.14.4
cloudpickle         1.6.0
colorama            0.4.4
cycler              0.10.0
cython_runtime      NA
cytoolz             0.11.2
dask                2022.01.0
dateutil            2.8.1
decorator           4.4.2
defusedxml          0.6.0
fasteners           0.17.3
fsspec              2022.01.0
google              NA
h5py                3.1.0
igraph              0.10.2
importlib_metadata  1.7.0
ipykernel           5.4.2
ipython_genutils    0.2.0
ipywidgets          7.6.3
jedi                0.17.2
jinja2              3.1.2
joblib              1.0.0
kiwisolver          1.3.1
leidenalg           0.9.0
llvmlite            0.35.0
louvain             0.7.0
markupsafe          2.1.1
mpl_toolkits        NA
msgpack             1.0.3
natsort             7.1.0
nbinom_ufunc        NA
numba               0.52.0
numcodecs           0.9.1
numexpr             2.7.2
packaging           21.3
parso               0.7.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
plotly              5.10.0
prompt_toolkit      3.0.10
psutil              5.9.0
ptyprocess          0.7.0
pyarrow             9.0.0
pygments            2.13.0
pyparsing           2.4.7
pytz                2020.5
scipy               1.7.3
setuptools_scm      NA
sinfo               0.3.1
six                 1.14.0
sklearn             1.0.2
sphinxcontrib       NA
statsmodels         0.12.1
storemagic          NA
tables              3.6.1
tblib               1.7.0
texttable           1.6.3
threadpoolctl       2.1.0
tlz                 0.11.2
toolz               0.11.2
tornado             6.1
traitlets           5.0.5
typing_extensions   NA
wcwidth             0.2.5
yaml                5.4.1
zarr                2.11.3
zipp                NA
zmq                 21.0.1
zope                NA
```

Example 5 (unknown):
```unknown
-----
IPython             7.19.0
jupyter_client      6.1.11
jupyter_core        4.7.0
notebook            6.2.0
-----
Python 3.7.9 | packaged by conda-forge | (default, Dec  9 2020, 21:08:20) [GCC 9.3.0]
Linux-4.15.0-112-generic-x86_64-with-debian-bullseye-sid
-----
Session information updated at 2023-02-08 21:05
```

Example 6 (python):
```python
sc.settings.set_figure_params(dpi=120)
```

Example 7 (unknown):
```unknown
multiplexed_samples = ['7089STDY13216920_BHF_F_Hea13242527','7089STDY13216921_BHF_F_Hea13242528',
                       '7089STDY13216922_BHF_F_Hea13242529','7089STDY13216923_BHF_F_Hea13242530']
```

Example 8 (python):
```python
# read in
soc = pd.read_csv('/nfs/team205/heart/mapped/cellranger-arc200/SOC_out/SOC-clusters_multiome-foetal_20221221.csv',
                 index_col=0)
print(soc.shape)
soc.head()
```

Example 9 (unknown):
```unknown
(74984, 10)
```

Example 10 (unknown):
```unknown
soc['donor_assignment'].value_counts()
```

Example 11 (unknown):
```unknown
Hst39               36323
Hst40               20246
donor_unassigned    13322
Hst36                5093
Name: donor_assignment, dtype: int64
```

Example 12 (python):
```python
soc_assigned_obsnames = list(soc.index[soc['donor_assignment']!='donor_unassigned'])
soc_unassigned_obsnames = list(soc.index[soc['donor_assignment']=='donor_unassigned'])
print(f'SOC assigned: {len(soc_assigned_obsnames)}')
print(f'SOC unassigned: {len(soc_unassigned_obsnames)}')
```

Example 13 (unknown):
```unknown
SOC assigned: 61662
SOC unassigned: 13322
```

Example 14 (python):
```python
# read in
cmo = pd.read_csv('/nfs/team205/heart/mapped/CMO/result_summary/CMO-result_multiome-foetal_20230207.csv',
                 index_col=0)
# extract soc-unassigned nuclei
cmo = cmo.loc[soc_unassigned_obsnames]

print(cmo.shape)
cmo.head()
```

Example 15 (python):
```python
# assign donors which has prop_delta > 2.4282365
# refer to notebook No3
cmo_assigned_obsnames = list(cmo.index[cmo['ratio'] > 2.4282365])
print(f'CMO assigned: {len(cmo_assigned_obsnames)}')

# change assignment to "unassigned" for the nuclei which have "prop_delta" lower than the threshold
cmo.loc[[x for x in cmo.index if x not in cmo_assigned_obsnames],'donor_assignment'] = 'donor_unassigned'
cmo['donor_assignment'].value_counts()
```

Example 16 (unknown):
```unknown
CMO assigned: 1885
```

Example 17 (unknown):
```unknown
donor_unassigned    11437
Hst40                1690
Hst36                 155
Hst39                  40
Name: donor_assignment, dtype: int64
```

Example 18 (unknown):
```unknown
# SOC
col_to_add = ['status','log_prob_singleton','log_prob_doublet', 'donor_assignment']
summary = soc[col_to_add].copy()
summary.columns = [f'SOC | {x}' for x in summary.columns]
summary.head()
```

Example 19 (python):
```python
# make final assignment column by adding CMO result
summary.rename(columns={'SOC | donor_assignment':'SOC-CMO_assignment'},inplace=True)
summary.loc[cmo.index,'SOC-CMO_assignment'] = cmo['donor_assignment'].astype('str')
print(summary['SOC-CMO_assignment'].value_counts())
summary.head()
```

Example 20 (unknown):
```unknown
Hst39               36363
Hst40               21936
donor_unassigned    11437
Hst36                5248
Name: SOC-CMO_assignment, dtype: int64
```

Example 21 (unknown):
```unknown
# save
summary.to_csv('/nfs/team205/heart/mapped/cellranger-arc200/SOC_out/SOC-CMO-assignment_multiome-foetal_20230207.csv')
```

---
