Utilities¶

dagster.file_relative_path(dunderfile, relative_path)[source]¶

Get a path relative to the currently executing Python file.

This function is useful when one needs to load a file that is relative to the position of the current file. (Such as when you encode a configuration file path in source file and want in runnable in any current working directory)

Parameters

• dunderfile (str) – Should always be __file__.

• relative_path (str) – Path to get relative to the currently executing file.

Examples:

file_relative_path(__file__, 'path/relative/to/file')

dagster.config_from_files(config_files)[source]¶

Constructs run config from YAML files.

Parameters

config_files (List[str]) – List of paths or glob patterns for yaml files to load and parse as the run config.

Returns

A run config dictionary constructed from provided YAML files.

Return type

Dict[str, Any]

Raises

• FileNotFoundError – When a config file produces no results

• DagsterInvariantViolationError – When one of the YAML files is invalid and has a parse error.

dagster.config_from_pkg_resources(pkg_resource_defs)[source]¶

Load a run config from a package resource, using pkg_resources.resource_string().

Example:

config_from_pkg_resources(
    pkg_resource_defs=[
        ('dagster_examples.airline_demo.environments', 'local_base.yaml'),
        ('dagster_examples.airline_demo.environments', 'local_warehouse.yaml'),
    ],
)

Parameters

pkg_resource_defs (List[(str, str)]) – List of pkg_resource modules/files to load as the run config.

Returns

A run config dictionary constructed from the provided yaml strings

Return type

Dict[Str, Any]

Raises

DagsterInvariantViolationError – When one of the YAML documents is invalid and has a parse error.

dagster.config_from_yaml_strings(yaml_strings)[source]¶

Static constructor for run configs from YAML strings.

Parameters

yaml_strings (List[str]) – List of yaml strings to parse as the run config.

Returns

A run config dictionary constructed from the provided yaml strings

Return type

Dict[Str, Any]

Raises

DagsterInvariantViolationError – When one of the YAML documents is invalid and has a parse error.

dagster.get_dagster_logger(name=None)[source]¶

Creates a python logger whose output messages will be captured and converted into Dagster log messages. This means they will have structured information such as the step_key, run_id, etc. embedded into them, and will show up in the Dagster event log.

This can be used as a more convenient alternative to context.log in most cases. If log level is not set explicitly, defaults to DEBUG.

Parameters

name (Optional[str]) – If supplied, will create a logger with the name “dagster.builtin.{name}”, with properties inherited from the base Dagster logger. If omitted, the returned logger will be named “dagster.builtin”.

Returns

A logger whose output will be captured by Dagster.

Return type

logging.Logger

Example

from dagster import get_dagster_logger, op

@op
def hello_op():
    log = get_dagster_logger()
    for i in range(5):
        # do something
        log.info(f"Did {i+1} things!")

class dagster.ExperimentalWarning[source]¶

class dagster.utils.forked_pdb.ForkedPdb(completekey='tab', stdin=None, stdout=None, skip=None, nosigint=False, readrc=True)[source]¶

A pdb subclass that may be used from a forked multiprocessing child

Examples:

from dagster.utils.forked_pdb import ForkedPdb

@solid
def complex_solid(_):
    # some complicated stuff

    ForkedPdb().set_trace()

    # some other complicated stuff

You can initiate pipeline execution via dagit and use the pdb debugger to examine/step through execution at the breakpoint.

dagster.utils.make_email_on_run_failure_sensor(*args, **kwargs)[source]¶