The alkymi API¶
Decorators¶
- alkymi.decorators.foreach(mapped_inputs: alkymi.recipe.Recipe, ingredients=(), name: Optional[str] = None, transient: bool = False, cache: alkymi.config.CacheType = CacheType.Auto) Callable[[Callable[[...], alkymi.decorators.R]], alkymi.foreach_recipe.ForeachRecipe[alkymi.decorators.R]] [source]¶
Convert a function into an alkymi Recipe to enable caching and conditional evaluation
- Parameters
mapped_inputs – A single Recipe to whose output (a list or dictionary) the bound function will be applied to generate the new outputs (similar to Python’s built-in map() function)
ingredients – The dependencies of this Recipe - the outputs of these Recipes will be provided as arguments to the bound function when called in the order that they were provided. If not all arguments are provided directly, alkymi will look up recipes that match the name of arguments automatically
name – The name to assign to the created recipe - if not provided, the bound function’s name will be used
transient – Whether to always (re)evaluate the created Recipe
cache – The type of caching to use for this Recipe
- Returns
A callable that will yield the Recipe created from the bound function
- alkymi.decorators.recipe(ingredients=(), name: Optional[str] = None, transient: bool = False, cache: alkymi.config.CacheType = CacheType.Auto) Callable[[Callable[[...], alkymi.decorators.R]], alkymi.recipe.Recipe[alkymi.decorators.R]] [source]¶
Convert a function into an alkymi Recipe to enable caching and conditional evaluation
- Parameters
ingredients – The dependencies of this Recipe - the outputs of these Recipes will be provided as arguments to the bound function when called in the order that they were provided. If not all arguments are provided directly, alkymi will look up recipes that match the name of arguments automatically
name – The name to assign to the created recipe - if not provided, the bound function’s name will be used
transient – Whether to always (re)evaluate the created Recipe
cache – The type of caching to use for this Recipe
- Returns
A callable that will yield the Recipe created from the bound function
Recipe¶
- class alkymi.recipe.Recipe(func: Callable[[...], alkymi.recipe.R], ingredients: Iterable[alkymi.recipe.Recipe], name: str, transient: bool, cache: alkymi.config.CacheType, cleanliness_func: Optional[Callable[[alkymi.recipe.R], bool]] = None)[source]¶
Recipe is the basic building block of alkymi’s evaluation approach. It binds a function (provided by the user) that it then calls when asked to by alkymi’s execution engine. The result of the bound function evaluation can be automatically cached to disk to allow for checking of cleanliness (whether a Recipe is up-to-date), and to avoid invoking the bound function if necessary on subsequent evaluations
- brew() alkymi.recipe.R [source]¶
Evaluate this Recipe and all dependent inputs - this will build the computational graph and execute any needed dependencies to produce the outputs of this Recipe
- Returns
The outputs of this Recipe (which correspond to the outputs of the bound function)
- property cache: alkymi.config.CacheType¶
- Returns
The type of caching to use for this Recipe
- property function_hash: str¶
- Returns
The hash of the bound function as a string
- property ingredients: List[alkymi.recipe.Recipe]¶
- Returns
The dependencies of this Recipe - the outputs of these Recipes will be provided as arguments to the bound function when called (following the item from the mapped_inputs sequence)
- property input_checksums: Optional[Tuple[Optional[str], ...]]¶
- Returns
The checksum(s) for the inputs
- invoke(inputs: Tuple[Any, ...], input_checksums: Tuple[Optional[str], ...]) alkymi.recipe.R [source]¶
Evaluate this Recipe using the provided inputs. This will call the bound function on the inputs. If the result is already cached, that result will be used instead (the checksum is used to check this). Only the immediately previous invoke call will be cached
- Parameters
inputs – The inputs provided by the ingredients (dependencies) of this Recipe
input_checksums – The (possibly new) input checksum to use for checking cleanliness
- Returns
The outputs of this Recipe (which correspond to the outputs of the bound function)
- property name: str¶
- Returns
The name of this Recipe
- property output_checksum: Optional[str]¶
- Returns
The computed checksums for the outputs (this is set when outputs are set)
- property outputs: Optional[alkymi.recipe.R]¶
- Returns
The outputs of this Recipe
- property outputs_valid: bool¶
Check whether an output is still valid - this is currently only used to check files that may have been deleted or altered outside alkymi’s cache. If no outputs have been produced yet, True will be returned.
- Returns
Whether all outputs are still valid
- restore_from_dict(old_state) None [source]¶
Restores the state of this Recipe from a previously cached state
- Parameters
old_state – The old cached state to restore
- status() alkymi.types.Status [source]¶
- Returns
The status of this recipe (will evaluate all upstream dependencies)
- property transient: bool¶
- Returns
Whether to always (re)evaluate the created Recipe
ForeachRecipe¶
- class alkymi.foreach_recipe.ForeachRecipe(mapped_recipe: alkymi.recipe.Recipe, ingredients: Iterable[alkymi.recipe.Recipe], func: Callable[[...], alkymi.foreach_recipe.R], name: str, transient: bool, cache: alkymi.config.CacheType, cleanliness_func: Optional[Callable[[alkymi.recipe.R], bool]] = None)[source]¶
Special type of Recipe that applies its bound function to each input from a list or dictionary (similar to Python’s built-in map() function). Evaluations of the bound function are cached and used to avoid reevaluation previously seen inputs, this means that changing the inputs to a ForeachRecipe may only trigger reevaluation of the bound function for some inputs, avoiding the overhead of recomputing things
- invoke_mapped(mapped_inputs: Union[List[Any], Dict[Any, Any]], mapped_inputs_checksum: Optional[str], inputs: Tuple[Any, ...], input_checksums: Tuple[Optional[str], ...]) Optional[Union[List[alkymi.serialization.Output], Dict[Any, alkymi.serialization.Output]]] [source]¶
Evaluate this ForeachRecipe using the provided inputs. This will apply the bound function to each item in the “mapped_inputs”. If the result for any item is already cached, that result will be used instead (the checksum is used to check this). Only items from the immediately previous invoke call will be cached
- Parameters
mapped_inputs – The (possibly new) sequence of inputs to apply the bound function to
mapped_inputs_checksum – A single checksum for all the mapped inputs, used to quickly check whether anything has changed
inputs – The inputs provided by the ingredients (dependencies) of this ForeachRecipe
input_checksums – The (possibly new) input checksum to use for checking cleanliness
- Returns
The outputs of this ForeachRecipe
- property mapped_inputs: Optional[Union[List[Any], Dict[Any, Any]]]¶
- Returns
The sequence of inputs to apply the bound function to
- property mapped_inputs_checksum: Optional[str]¶
- Returns
The summary of the mapped inputs checksum
- property mapped_inputs_checksums: Optional[Union[List[Optional[str]], Dict[Any, Optional[str]]]]¶
- Returns
The computed checksums for the sequence of mapped inputs (this is set when mapped_inputs is set)
- property mapped_outputs_checksums: Optional[Union[List[Optional[str]], Dict[Any, Optional[str]]]]¶
- Returns
The computed checksums for the sequence of mapped outputs
- property mapped_recipe: alkymi.recipe.Recipe¶
- Returns
The dependent Recipe that produces the input sequence to map the bound function to
- property output_checksum: Optional[str]¶
- Returns
The computed checksums for the outputs (this is set when outputs are set)
- property outputs: Optional[Union[Dict, List]]¶
- Returns
The outputs of this ForeachRecipe in canonical form (None or a tuple with zero or more entries)
- property outputs_valid: bool¶
Check whether an output is still valid - this is currently only used to check files that may have been deleted or altered outside of alkymi’s cache. If no outputs have been produced yet, True will be returned.
- Returns
Whether all outputs are still valid
Checksums¶
- class alkymi.checksums.Checksummer[source]¶
Class used to compute a stable hash/checksum of an object recursively. Currently uses MD5.
Serialization¶
- class alkymi.serialization.CachedOutput(value: Optional[alkymi.serialization.T], checksum: str, serializable_representation: Union[str, int, float, None, Iterable[Optional[Union[str, int, float]]], Dict[str, Optional[Union[str, int, float]]]])[source]¶
An Output that has been cached - may or may not have it’s associated value in-memory
- property serialized: Union[str, int, float, None, Iterable[Optional[Union[str, int, float]]], Dict[str, Optional[Union[str, int, float]]]]¶
- Returns
A serializable representation of the value of this output
- property valid: bool¶
- Returns
Whether this Output is still valid (e.g. an external file pointed to by a Path can have been altered)
- class alkymi.serialization.Output(checksum: str)[source]¶
Abstract base class for keeping track of outputs of Recipes
- property checksum: str¶
- Returns
The checksum of this Output
- property valid: bool¶
- Returns
Whether this Output is still valid (e.g. an external file pointed to by a Path can have been altered)
- class alkymi.serialization.OutputWithValue(value: alkymi.serialization.T, checksum: str)[source]¶
An Output that is guaranteed to have an in-memory value - all outputs start out as this before being cached
- property valid: bool¶
- Returns
Whether this Output is still valid (e.g. an external file pointed to by a Path can have been altered)
- class alkymi.serialization.Serializer(*args, **kwds)[source]¶
Abstract base class for classes that enable serialization/deserialization of classes not in the standard library
- alkymi.serialization.cache(output: alkymi.serialization.OutputWithValue, base_path: pathlib.Path) alkymi.serialization.CachedOutput [source]¶
Cache an in-memory OutputWithValue, thus converting it to a CachedOutput. The resulting output will retain the value in-memory
- Parameters
output – The Output to cache
base_path – The directory to use for this serialization. A subdirectory will be created to store complex serialized objects
- Returns
The cached output
- alkymi.serialization.create_token(name) str [source]¶
Creates a token using the ‘TOKEN_TEMPLATE’ to signify to the deserializer func how to deserializer a given value
- Parameters
name – The name of the token
- Returns
A new token for the given token name
- alkymi.serialization.deserialize_item(item: Union[str, int, float, None, Iterable[Optional[Union[str, int, float]]], Dict[str, Optional[Union[str, int, float]]]]) Any [source]¶
Deserializes an item (potentially recursively)
- Parameters
item – The item to deserialize (may be nested)
- Returns
The deserialized item
- alkymi.serialization.from_cache(serializable_representation: Union[str, int, float, None, Iterable[Optional[Union[str, int, float]]], Dict[str, Optional[Union[str, int, float]]]]) Any [source]¶
Deserialize an output from the cache using its serialized representation
- Parameters
serializable_representation – The serialized representation to deserialize
- Returns
The deserialized object
- alkymi.serialization.is_valid_serialized(item: Union[str, int, float, None, Iterable[Optional[Union[str, int, float]]], Dict[str, Optional[Union[str, int, float]]]]) bool [source]¶
Recursively check validity of a serialized representation. Currently just looks for external files represented by Path objects, and then compares the stored checksum of each such item with the current checksum (computed from the current file contents)
- Parameters
item – The serialized representation to check validity for
- Returns
True if the input is still valid
- alkymi.serialization.serialize_item(item: Any, cache_path_generator: Generator[pathlib.Path, None, None]) Union[str, int, float, None, Iterable[Optional[Union[str, int, float]]], Dict[str, Optional[Union[str, int, float]]]] [source]¶
Serializes an item (potentially recursively)
- Parameters
item – The item to serialize (may be nested)
cache_path_generator – The generator to use for generating cache paths
- Returns
The serialized item
Lab¶
- class alkymi.lab.Lab(name: str)[source]¶
Class used to define a collection of alkymi recipes and expose them as a command line interface (CLI)
This can be used to create files that bear resemblance to Makefiles (see alkymi/labfile.py as an example)
- add_recipe(recipe: alkymi.recipe.Recipe) alkymi.recipe.Recipe [source]¶
Add a new recipe to the Lab (this will make the recipe available through the CLI)
- Parameters
recipe – The recipe to add
- Returns
The input recipe (to allow chaining calls)
- add_recipes(*recipes: alkymi.recipe.Recipe) None [source]¶
Add a set of recipes to the Lab (this will make the recipes available through the CLI)
- Parameters
recipes – The recipes to add
- property args: Dict[str, alkymi.recipes.Arg]¶
- Returns
The list of args registered with this Lab
- brew(target_recipe: Union[alkymi.recipe.Recipe, str]) Any [source]¶
Brew (evaluate) a target recipe defined by its reference or name, and return the results
- Parameters
target_recipe – The recipe to evaluate, as a reference ot by name
- Returns
The output of the evaluated recipe
- property name: str¶
- Returns
The name of this Lab
- open(args: Optional[List[str]] = None, stream: TextIO = <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>) None [source]¶
Runs the command line interface for this Lab by parsing command line arguments and carrying out the designated command
- Parameters
args – The input arguments to use - will default to system args
stream – The stream to print output to
- property recipes: List[alkymi.recipe.Recipe]¶
- Returns
The list of recipes contained in this Lab
- register_arg(arg: alkymi.recipes.Arg) None [source]¶
Register an argument with the Lab (this will make the argument settable through the CLI)
- Parameters
arg – The argument to register
Built-in Recipes¶
- class alkymi.recipes.Arg(arg: alkymi.recipes.T, name: str, cache=CacheType.Auto)[source]¶
Class providing a stateful argument
To use, create an Arg instance with the initial value for your argument, e.g.
Arg(0)
, then provide as a recipe to downstream recipes. To change the input arguments, callset()
again - this will mark the recipe as dirty and cause reevaluation of downstream recipe(s)- set(_arg: alkymi.recipes.T) None [source]¶
Change the argument, causing the recipe to need reevaluation
- Parameters
_arg – The new argument
- property subtype: Optional[Any]¶
- Returns
The subtype of the argument (e.g. the type of items contained in a list). Will be None for non-iterable types
- property type: Type[alkymi.recipes.T]¶
- Returns
The type of the argument
- alkymi.recipes.arg(_arg: alkymi.recipes.T, name: str, cache=CacheType.Auto) alkymi.recipes.Arg[alkymi.recipes.T] [source]¶
Shorthand for creating an
Arg
instance- Parameters
_arg – The initial argument to use
name – The name to give the created Recipe
cache – The type of caching to use for this Recipe
- Returns
The created
Arg
instance
- alkymi.recipes.file(name: str, path: pathlib.Path, cache=CacheType.Auto) alkymi.recipe.Recipe[pathlib.Path] [source]¶
Create a Recipe that outputs a single file
- Parameters
name – The name to give the created Recipe
path – The path to the file to output
cache – The type of caching to use for this Recipe
- Returns
The created Recipe
- alkymi.recipes.glob_files(name: str, directory: pathlib.Path, pattern: str, recursive: bool, cache=CacheType.Auto) alkymi.recipe.Recipe[List[pathlib.Path]] [source]¶
Create a Recipe that will glob files in a directory and return them as a list. The created recipe will only be considered dirty if the file paths contained in the glob changes (not if the contents of any one file changes)
- Parameters
name – The name to give the created Recipe
directory – The directory in which to perform the glob
pattern – The pattern to use for the glob, e.g. *.py
recursive – Whether to glob recursively into subdirectories
cache – The type of caching to use for this Recipe
- Returns
The created Recipe
- alkymi.recipes.zip_results(name: str, recipes: Iterable[alkymi.recipe.Recipe], cache=CacheType.Auto) alkymi.recipe.Recipe[Union[List[Tuple[Any, ...]], Dict[Any, Tuple[Any, ...]]]] [source]¶
Create a Recipe that zips the outputs from a number of recipes into elements, similar to Python’s built-in zip(). Notably, dictionaries are handled a bit differently, in that a dictionary is returned with keys mapping to tuples from the different inputs, i.e.:
{"1": 1} zip {"1", "one"} -> {"1", (1, "one")}
- Parameters
name – The name to give the created Recipe
recipes – The recipes to zip. These must return lists or dictionaries
cache – The type of caching to use for this Recipe
- Returns
The created Recipe
Core¶
- alkymi.alkymi.brew(recipe: alkymi.recipe.Recipe[alkymi.recipe.R]) alkymi.recipe.R [source]¶
Evaluate a Recipe and all dependent inputs - this will build the computational graph and execute any needed dependencies to produce the outputs of the input Recipe
- Parameters
recipe – The Recipe to evaluate
- Returns
The outputs of the Recipe (which correspond to the outputs of the bound function)
- alkymi.alkymi.compute_recipe_status(recipe: alkymi.recipe.Recipe[alkymi.recipe.R]) Dict[alkymi.recipe.Recipe, alkymi.types.Status] [source]¶
Compute the Status for the provided recipe and all dependencies (ingredients or mapped inputs)
- Parameters
recipe – The recipe for which status should be computed
- Returns
The status of the provided recipe and all dependencies as a dictionary
- alkymi.alkymi.compute_status_with_cache(recipe: alkymi.recipe.Recipe[alkymi.recipe.R], status: Dict[alkymi.recipe.Recipe, alkymi.types.Status]) alkymi.types.Status [source]¶
Compute the Status for the provided recipe and all dependencies (ingredients or mapped inputs) and store the results in the provided status dictionary.
This function will early-exit if the status for the provided recipe has already been computed, and will recursively compute statuses of dependencies (ingredients or mapped inputs)
- Parameters
recipe – The recipe for which status should be computed
status – The dictionary to add computed statuses to
- Returns
The status of the input recipe
- alkymi.alkymi.evaluate_recipe(recipe: alkymi.recipe.Recipe[alkymi.recipe.R], status: Dict[alkymi.recipe.Recipe, alkymi.types.Status]) Tuple[alkymi.recipe.R, Optional[str]] [source]¶
Evaluate a recipe using precomputed statuses
- Parameters
recipe – The recipe to evaluate
status – The dictionary of statuses computed using ‘compute_recipe_status()’ to use for targeted evaluation
- Returns
The outputs and checksums of the provided recipe
- alkymi.alkymi.is_clean(recipe: alkymi.recipe.Recipe[alkymi.recipe.R], new_input_checksums: Tuple[Optional[str], ...]) alkymi.types.Status [source]¶
Check whether a Recipe is clean (result is cached) based on a set of (potentially new) input checksums
- Parameters
recipe – The Recipe to check for cleanliness
new_input_checksums – The (potentially new) input checksums to use for checking cleanliness
- Returns
Whether the recipe is clean represented by the Status enum
- alkymi.alkymi.is_foreach_clean(recipe: alkymi.foreach_recipe.ForeachRecipe[alkymi.recipe.R], mapped_inputs_checksum: Optional[str]) bool [source]¶
Check whether a ForeachRecipe is clean (in addition to the regular recipe cleanliness checks). This is done by comparing the overall checksum for the current mapped inputs to that from the last invoke evaluation
- Parameters
recipe – The ForeachRecipe to check cleanliness of mapped inputs for
mapped_inputs_checksum – A single checksum for all the mapped inputs, used to quickly check whether anything has changed
- Returns
Whether the input recipe needs to be reevaluated
AlkymiConfig¶
- class alkymi.config.AlkymiConfig[source]¶
Global singleton config for alkymi
- property allow_pickling: bool¶
- Returns
Whether to allow pickling for serialization, deserialization and checksumming
- property cache: bool¶
- Returns
Whether to enable alkymi caching globally (see CacheType.Auto)
- property cache_path: Optional[pathlib.Path]¶
- Returns
A user-provided location to place the cache
- property file_checksum_method: alkymi.config.FileChecksumMethod¶
- Returns
The currently used method for calculating file checksums (for Path objects)
- static get() alkymi.config.AlkymiConfig [source]¶
- Returns
The singleton instance of AlkymiConfig