Analysis flow#

Here, we’ll track typical data transformations like subsetting that occur during analysis.

If exploring more generally, read this first: Project flow.

Setup#

# a lamindb instance containing Bionty schema
!lamin init --storage ./analysis-usecase --schema bionty

import lamindb as ln
import bionty as bt
from lamin_utils import logger

bt.settings.auto_save_parents = False

💡 connected lamindb: testuser1/analysis-usecase

Register an initial dataset#

Here we register an initial artifact with a pipeline script register_example_file.py.

!python analysis-flow-scripts/register_example_file.py

Show code cell output Hide code cell output

💡 connected lamindb: testuser1/analysis-usecase

💡 saved: Transform(uid='K4wsS5DTYdFp6K79', name='register_example_file.py', key='register_example_file.py', version='0', type='script', updated_at=2024-04-24 12:50:52 UTC, created_by_id=1)
💡 saved: Run(uid='0OWgZJwxKGg4vDoKYaoS', transform_id=1, created_by_id=1)

✅ saved transform.source_code: Artifact(uid='thxLjOgCx7LjbrEvuTbA', suffix='.py', description='Source of transform K4wsS5DTYdFp6K79', version='0', size=699, hash='p9yfNIWwDKdfz8URlTmM0Q', hash_type='md5', visibility=0, key_is_virtual=True, updated_at=2024-04-24 12:50:52 UTC, storage_id=1, created_by_id=1)
✅ saved run.environment: Artifact(uid='DeqrVoZyvLUJlIL7kGkG', suffix='.txt', description='requirements.txt', size=3429, hash='zifvwWK3ZlTccmUo2WTBzw', hash_type='md5', visibility=0, key_is_virtual=True, updated_at=2024-04-24 12:50:52 UTC, storage_id=1, created_by_id=1)

✅ added 3 records with Feature.name for columns: ['cell_type', 'tissue', 'disease']

❗ 1 non-validated categories are not saved in Feature.name: ['cell_type_id']!
      → to lookup categories, use lookup().columns
      → to save, run add_new_from_columns

✅ added 99 records from public with Gene.ensembl_gene_id for var_index: ['ENSG00000000003', 'ENSG00000000005', 'ENSG00000000419', 'ENSG00000000457', 'ENSG00000000460', 'ENSG00000000938', 'ENSG00000000971', 'ENSG00000001036', 'ENSG00000001084', 'ENSG00000001167', 'ENSG00000001460', 'ENSG00000001461', 'ENSG00000001497', 'ENSG00000001561', 'ENSG00000001617', 'ENSG00000001626', 'ENSG00000001629', 'ENSG00000001630', 'ENSG00000001631', 'ENSG00000002016', 'ENSG00000002079', 'ENSG00000002330', 'ENSG00000002549', 'ENSG00000002586', 'ENSG00000002587', 'ENSG00000002726', 'ENSG00000002745', 'ENSG00000002746', 'ENSG00000002822', 'ENSG00000002834', 'ENSG00000002919', 'ENSG00000002933', 'ENSG00000003056', 'ENSG00000003096', 'ENSG00000003137', 'ENSG00000003147', 'ENSG00000003249', 'ENSG00000003393', 'ENSG00000003400', 'ENSG00000003402', 'ENSG00000003436', 'ENSG00000003509', 'ENSG00000003756', 'ENSG00000003987', 'ENSG00000003989', 'ENSG00000004059', 'ENSG00000004139', 'ENSG00000004142', 'ENSG00000004399', 'ENSG00000004455', 'ENSG00000004468', 'ENSG00000004478', 'ENSG00000004487', 'ENSG00000004534', 'ENSG00000004660', 'ENSG00000004700', 'ENSG00000004766', 'ENSG00000004776', 'ENSG00000004777', 'ENSG00000004779', 'ENSG00000004799', 'ENSG00000004809', 'ENSG00000004838', 'ENSG00000004846', 'ENSG00000004848', 'ENSG00000004864', 'ENSG00000004866', 'ENSG00000004897', 'ENSG00000004939', 'ENSG00000004948', 'ENSG00000004961', 'ENSG00000004975', 'ENSG00000005001', 'ENSG00000005007', 'ENSG00000005020', 'ENSG00000005022', 'ENSG00000005059', 'ENSG00000005073', 'ENSG00000005075', 'ENSG00000005100', 'ENSG00000005102', 'ENSG00000005108', 'ENSG00000005156', 'ENSG00000005175', 'ENSG00000005187', 'ENSG00000005189', 'ENSG00000005194', 'ENSG00000005206', 'ENSG00000005238', 'ENSG00000005243', 'ENSG00000005249', 'ENSG00000005302', 'ENSG00000005339', 'ENSG00000005379', 'ENSG00000005381', 'ENSG00000005421', 'ENSG00000005436', 'ENSG00000005448', 'ENSG00000005469']
💡 saving labels for 'cell_type'

✅ added 3 records from public with CellType.name for cell_type: ['T cell', 'hematopoietic stem cell', 'hepatocyte']
❗ 1 non-validated categories are not saved in CellType.name: ['my new cell type']!
      → to lookup categories, use lookup().cell_type
      → to save, run .add_new_from('cell_type')
💡 saving labels for 'tissue'

✅ added 4 records from public with Tissue.name for tissue: ['kidney', 'liver', 'heart', 'brain']
💡 saving labels for 'disease'

✅ added 4 records from public with Disease.name for disease: ['chronic kidney disease', 'liver lymphoma', 'cardiac ventricle disorder', 'Alzheimer disease']

✅ added 1 record with CellType.name for cell_type: ['my new cell type']
✅ var_index is validated against Gene.ensembl_gene_id
✅ cell_type is validated against CellType.name

✅ tissue is validated against Tissue.name
✅ disease is validated against Disease.name

💡 path content will be copied to default storage upon `save()` with key `None` ('.lamindb/46kSnPIeg4WVuPI56mp0.h5ad')
✅ storing artifact '46kSnPIeg4WVuPI56mp0' at '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/46kSnPIeg4WVuPI56mp0.h5ad'
💡 parsing feature names of X stored in slot 'var'
✅    99 terms (100.00%) are validated for ensembl_gene_id
✅    linked: FeatureSet(uid='p5yGYCYnAC24Q2lxSXHX', n=99, type='number', registry='bionty.Gene', hash='-frOq7J0bik-J7Ad9DX7', created_by_id=1)
💡 parsing feature names of slot 'obs'
✅    3 terms (75.00%) are validated for name
❗    1 term (25.00%) is not validated for name: cell_type_id
✅    linked: FeatureSet(uid='cGzhwEXE7NoeJhU7N0qI', n=3, registry='core.Feature', hash='tx0AhZiCU-u0NbXb7CGJ', created_by_id=1)

✅ saved 2 feature sets for slots: 'var','obs'
✅ linked feature 'cell_type' to registry 'bionty.CellType'
✅ linked feature 'tissue' to registry 'bionty.Tissue'

✅ linked feature 'disease' to registry 'bionty.Disease'

Pull the registered dataset, apply a transformation, and register the result#

Track the current notebook:

ln.settings.transform.stem_uid = "eNef4Arw8nNM"
ln.settings.transform.version = "0"
ln.track()

💡 notebook imports: bionty==0.42.9 lamin_utils==0.13.2 lamindb==0.70.4

💡 saved: Transform(uid='eNef4Arw8nNM6K79', name='Analysis flow', key='analysis-flow', version='0', type='notebook', updated_at=2024-04-24 12:52:04 UTC, created_by_id=1)

💡 saved: Run(uid='hWZZha1gBVxIJLCxd01g', transform_id=2, created_by_id=1)

artifact = ln.Artifact.filter(description="anndata with obs").one()
artifact.describe()

Artifact(uid='46kSnPIeg4WVuPI56mp0', suffix='.h5ad', accessor='AnnData', description='anndata with obs', size=46992, hash='IJORtcQUSS11QBqD-nTD0A', hash_type='md5', n_observations=40, visibility=1, key_is_virtual=True, updated_at=2024-04-24 12:52:03 UTC)

Provenance:
  📎 storage: Storage(uid='CEf68atJ', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase', type='local')
  📎 transform: Transform(uid='K4wsS5DTYdFp6K79', name='register_example_file.py', key='register_example_file.py', version='0', type='script')
  📎 run: Run(uid='0OWgZJwxKGg4vDoKYaoS', started_at=2024-04-24 12:50:52 UTC, finished_at=2024-04-24 12:52:03 UTC)
  📎 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1')
Features:
  var: FeatureSet(uid='p5yGYCYnAC24Q2lxSXHX', n=99, type='number', registry='bionty.Gene')
    'KLHL13', 'SLC25A5', 'THSD7A', 'KRIT1', 'ABCB5', 'HS3ST1', 'SLC7A2', 'COPZ2', 'SARM1', 'RAD52', 'ALS2', 'ST7', 'NFYA', 'M6PR', 'NDUFAB1', 'AOC1', 'HSPB6', 'SKAP2', 'ARX', 'ARHGAP33', ...
  obs: FeatureSet(uid='cGzhwEXE7NoeJhU7N0qI', n=3, registry='core.Feature')
    🔗 cell_type (4, bionty.CellType): 'hematopoietic stem cell', 'hepatocyte', 'T cell', 'my new cell type'
    🔗 tissue (4, bionty.Tissue): 'heart', 'liver', 'kidney', 'brain'
    🔗 disease (4, bionty.Disease): 'Alzheimer disease', 'cardiac ventricle disorder', 'liver lymphoma', 'chronic kidney disease'
Labels:
  📎 tissues (4, bionty.Tissue): 'heart', 'liver', 'kidney', 'brain'
  📎 cell_types (4, bionty.CellType): 'hematopoietic stem cell', 'hepatocyte', 'T cell', 'my new cell type'
  📎 diseases (4, bionty.Disease): 'Alzheimer disease', 'cardiac ventricle disorder', 'liver lymphoma', 'chronic kidney disease'

Get a backed AnnData object#

adata = artifact.backed()
adata

AnnDataAccessor object with n_obs × n_vars = 40 × 100
  constructed for the AnnData object 46kSnPIeg4WVuPI56mp0.h5ad
    obs: ['_index', 'cell_type', 'cell_type_id', 'disease', 'tissue']
    var: ['_index']

Subset dataset to specific cell types and diseases#

cell_types = artifact.cell_types.all().lookup(return_field="name")
diseases = artifact.diseases.all().lookup(return_field="name")

Create the subset:

subset_obs = adata.obs.cell_type.isin(
    [cell_types.t_cell, cell_types.hematopoietic_stem_cell]
) & (adata.obs.disease.isin([diseases.liver_lymphoma, diseases.chronic_kidney_disease]))

adata_subset = adata[subset_obs]
adata_subset

AnnDataAccessorSubset object with n_obs × n_vars = 20 × 100
  obs: ['_index', 'cell_type', 'cell_type_id', 'disease', 'tissue']
  var: ['_index']

adata_subset.obs[["cell_type", "disease"]].value_counts()

cell_type                disease               
T cell                   chronic kidney disease    10
hematopoietic stem cell  liver lymphoma            10
dtype: int64

annotate = ln.Annotate.from_anndata(
    adata_subset.to_memory(), 
    var_index=bt.Gene.ensembl_gene_id, 
    categoricals={
        "cell_type": bt.CellType.name, 
        "disease": bt.Disease.name, 
        "tissue": bt.Tissue.name,
    },
    organism="human"
)

annotate.validate()

artifact = annotate.save_artifact(description="anndata with obs subset")

artifact.describe()

Artifact(uid='d6fFIGkqXkCXhAmhgQnm', suffix='.h5ad', accessor='AnnData', description='anndata with obs subset', size=38992, hash='RgGUx7ndRplZZSmalTAWiw', hash_type='md5', n_observations=20, visibility=1, key_is_virtual=True, updated_at=2024-04-24 12:52:04 UTC)

Provenance:
  📎 storage: Storage(uid='CEf68atJ', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase', type='local')
  📎 transform: Transform(uid='eNef4Arw8nNM6K79', name='Analysis flow', key='analysis-flow', version='0', type='notebook')
  📎 run: Run(uid='hWZZha1gBVxIJLCxd01g', started_at=2024-04-24 12:52:04 UTC, is_consecutive=True)
  📎 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1')
Features:
  var: FeatureSet(uid='p5yGYCYnAC24Q2lxSXHX', n=99, type='number', registry='bionty.Gene')
    'KLHL13', 'SLC25A5', 'THSD7A', 'KRIT1', 'ABCB5', 'HS3ST1', 'SLC7A2', 'COPZ2', 'SARM1', 'RAD52', 'ALS2', 'ST7', 'NFYA', 'M6PR', 'NDUFAB1', 'AOC1', 'HSPB6', 'SKAP2', 'ARX', 'ARHGAP33', ...
  obs: FeatureSet(uid='cGzhwEXE7NoeJhU7N0qI', n=3, registry='core.Feature')
    🔗 cell_type (2, bionty.CellType): 'hematopoietic stem cell', 'T cell'
    🔗 tissue (2, bionty.Tissue): 'liver', 'kidney'
    🔗 disease (2, bionty.Disease): 'liver lymphoma', 'chronic kidney disease'
Labels:
  📎 tissues (2, bionty.Tissue): 'liver', 'kidney'
  📎 cell_types (2, bionty.CellType): 'hematopoietic stem cell', 'T cell'
  📎 diseases (2, bionty.Disease): 'liver lymphoma', 'chronic kidney disease'

Examine data flow#

Query a subsetted .h5ad artifact containing “hematopoietic stem cell” and “T cell”:

cell_types = bt.CellType.lookup()

my_subset = ln.Artifact.filter(
    suffix=".h5ad",
    description__endswith="subset",
    cell_types__in=[
        cell_types.hematopoietic_stem_cell,
        cell_types.t_cell,
    ],
).first()

my_subset

Artifact(uid='d6fFIGkqXkCXhAmhgQnm', suffix='.h5ad', accessor='AnnData', description='anndata with obs subset', size=38992, hash='RgGUx7ndRplZZSmalTAWiw', hash_type='md5', n_observations=20, visibility=1, key_is_virtual=True, updated_at=2024-04-24 12:52:04 UTC, storage_id=1, transform_id=2, run_id=2, created_by_id=1)

Common questions that might arise are:

What is the history of this artifact?
Which features and labels are associated with it?
Which notebook analyzed and registered this artifact?
By whom?
And which artifact is its parent?

Let’s answer this using LaminDB:

print("--> What is the history of this artifact?\n")
artifact.view_lineage()

print("\n\n--> Which features and labels are associated with it?\n")
logger.print(artifact.features)
logger.print(artifact.labels)

print("\n\n--> Which notebook analyzed and registered this artifact\n")
logger.print(artifact.transform)

print("\n\n--> By whom\n")
logger.print(artifact.created_by)

print("\n\n--> And which artifact is its parent\n")
display(artifact.run.input_artifacts.df())

--> What is the history of this artifact?

_images/e31f5eb6e134b3750044994dfc70f1650122ae7133b6129e3fcbcccc8500fe6f.svg

--> Which features and labels are associated with it?

Features:
  var: FeatureSet(uid='p5yGYCYnAC24Q2lxSXHX', n=99, type='number', registry='bionty.Gene')
    'KLHL13', 'SLC25A5', 'THSD7A', 'KRIT1', 'ABCB5', 'HS3ST1', 'SLC7A2', 'COPZ2', 'SARM1', 'RAD52', 'ALS2', 'ST7', 'NFYA', 'M6PR', 'NDUFAB1', 'AOC1', 'HSPB6', 'SKAP2', 'ARX', 'ARHGAP33', ...
  obs: FeatureSet(uid='cGzhwEXE7NoeJhU7N0qI', n=3, registry='core.Feature')
    🔗 cell_type (2, bionty.CellType): 'hematopoietic stem cell', 'T cell'
    🔗 tissue (2, bionty.Tissue): 'liver', 'kidney'
    🔗 disease (2, bionty.Disease): 'liver lymphoma', 'chronic kidney disease'

Labels:
  📎 tissues (2, bionty.Tissue): 'liver', 'kidney'
  📎 cell_types (2, bionty.CellType): 'hematopoietic stem cell', 'T cell'
  📎 diseases (2, bionty.Disease): 'liver lymphoma', 'chronic kidney disease'

--> Which notebook analyzed and registered this artifact

Transform(uid='eNef4Arw8nNM6K79', name='Analysis flow', key='analysis-flow', version='0', type='notebook', updated_at=2024-04-24 12:52:04 UTC, created_by_id=1)

--> By whom

User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at=2024-04-24 12:50:49 UTC)

--> And which artifact is its parent

	uid	storage_id	key	suffix	accessor	description	version	size	hash	hash_type	n_objects	n_observations	transform_id	run_id	visibility	key_is_virtual	created_at	updated_at	created_by_id
id
3	46kSnPIeg4WVuPI56mp0	1	None	.h5ad	AnnData	anndata with obs	None	46992	IJORtcQUSS11QBqD-nTD0A	md5	None	40	1	1	1	True	2024-04-24 12:52:03.299313+00:00	2024-04-24 12:52:03.359832+00:00	1

Show code cell content Hide code cell content

!lamin delete --force analysis-usecase
!rm -r ./analysis-usecase

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.14/x64/bin/lamin", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/rich_click/rich_command.py", line 126, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamin_cli/__main__.py", line 103, in delete
    return delete(instance, force=force)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamindb_setup/_delete.py", line 130, in delete
    n_objects = check_storage_is_empty(
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/lamindb_setup/core/upath.py", line 720, in check_storage_is_empty
    raise InstanceNotEmpty(message)
lamindb_setup.core.upath.InstanceNotEmpty: Storage location contains 6 objects (2 ignored) - delete them prior to deleting the instance
['/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/46kSnPIeg4WVuPI56mp0.h5ad', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/DeqrVoZyvLUJlIL7kGkG.txt', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/_is_initialized', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/d6fFIGkqXkCXhAmhgQnm.h5ad', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/.lamindb/thxLjOgCx7LjbrEvuTbA.py', '/home/runner/work/lamin-usecases/lamin-usecases/docs/analysis-usecase/8c336e665ef65fe8a473562b5c712fba.lndb']