Overview of using apps#

There are 3 top-level functions that provide the major means for identifying what apps are installed, what an app can do and for getting an app to use it. These functions are:

available_apps() (see Displaying installed apps)
app_help() (see Getting help on an app)
get_app() (see Getting an app)

Two other crucial concepts concern data stores and tracking failures.

Types of apps#

There are 3 types of apps:

loaders (by convention, names starts with load_<data type>)
writers (by convention, names starts with write_<data type>)
generic (no naming convention)

As their names imply, loaders load, writers write and generic apps do other operations on data.

Composability#

Most cogent3 apps are “composable”, meaning that multiple apps can be combined into a single function by addition. For example, say we have an app (fit_model) that performs a molecular evolutionary analysis on an alignment, and another app (extract_stats) that gets the statistics from the result. We could perform these steps sequentially as follows

fitted = fit_model(alignment)
stats = extract_stats(fitted)

Composability allows us to simplify this as follows

app = fit_model + extract_stats
stats = app(fitted)

We can have many more apps in a composed function than just the two shown here.

Composability rules#

There are rules around app composition, starting with app types. Loaders and writers are special cases. If included, a loader must always be first, e.g.

app = a_loader + a_generic

If included, a writer must always be last, e.g.

app = a_generic + a_writer

Changing the order for either of the above will result in a TypeError.

The next constraint on app composition are the input and output types of the apps involved. Specifically, apps define the type of input they work on and the type of output they produce. For two apps to be composed, the output (or return) type of app on the left (e.g. a_loader) must overlap with the input type of the app on the right (e.g. a_generic). If they don’t match, a TypeError is raised.

An example#

I illustrate the general approach for a simple example – extracting third codon positions. As I’m defining a writer, I also need to define the destination (a directory in this case) where it will write to.

from cogent3 import get_app, open_data_store

out_dstore = open_data_store(path_to_dir, suffix="fa", mode="w")

loader = get_app("load_aligned", format="fasta", moltype="dna")
cpos3 = get_app("take_codon_positions", 3)
writer = get_app("write_seqs", out_dstore, format="fasta")

Using apps sequentially like functions#

data = loader("data/primate_brca1.fasta")
just3rd = cpos3(data)
m = writer(just3rd)

The resulting alignment just3rd will be written into the out_dstore directory in fasta format with the same filename as the original data ("primate_brca1.fasta").

Note

m is a DataMember of out_dstore.

Composing a multi-step process from several apps#

We can make this simpler by creating a single composed function.

process = loader + cpos3 + writer
m = process("data/primate_brca1.fasta")

Applying a process to multiple data records#

We use a data store to identify all data files in a directory that we want to analyse. process can be then applied to all records in the data store without having to loop.

dstore = open_data_store("data", suffix="fasta", mode="r")
result = process.apply_to(dstore)

Note

result is out_dstore.

Other important features#

The settings and data analysed will be logged#

A log file will be written into the same data store as the output. The log includes information on the conditions under which the analysis was run and fingerprint all input and output files.

out_dstore.summary_logs

summary of log files
time	name	python version	who	command	composable
2024-02-05 00:20:00	logs/load_aligned-take_codon_positions-write_seqs-74c2cbcc.log	3.12.1	runner	/home/runner/work/cogent3.github.io/cogent3.github.io/.venv/lib/python3.12/site-packages/ipykernel_launcher.py -f /tmp/tmp8jryr8ww.json --HistoryManager.hist_file=:memory:	load_aligned(moltype='dna', format='fasta') +take_codon_positions(positions=(3,), fourfold_degenerate=False, gc='Standard',moltype='dna') +write_seqs(data_store=DataStoreDirectory(source=/home/runner/work/cogent3.github.io/cogent3.github.io/c3org/doc/doc/tmp6odcoxja,mode=Mode.w, suffix=fa, limit=None, verbose=False), id_from_source=, format='fasta')

1 rows x 6 columns

Failures are recorded#

Any “failures” (see The NotCompleted object) are saved. The data store class provides methods for interrogating those. First, a general summary of the output data store indicates we have 6 records that did not complete.

out_dstore.describe

Directory datastore
record type	number
completed	8
not_completed	5
logs	1

3 rows x 2 columns

These occur for this example primarily because some of the files contain sequences that are not aligned

out_dstore.summary_not_completed

not completed records
type	origin	message	num	source
ERROR	load_aligned	"KeyError: 'F', Value... inhomogeneous part."	5	refseqs.fasta, SCA1-cds.fasta, ...

1 rows x 5 columns

You can track progress#

result = process.apply_to(dstore, show_progress=True)

You can do parallel computation#

result = process.apply_to(dstore, parallel=True)

By default, this will use all available processors on your machine. (See Parallel computations for more details plus how to take advantage of multiple machines using MPI.)

All of the above#

process.apply_to(dstore, parallel=True, show_progress=True)