Skip to main content

Schedule cron-based pipelines

Schedules enable automated execution of jobs at specified intervals. These intervals can range from common frequencies like hourly, daily, or weekly, to more intricate patterns defined using cron expressions.

Prerequisites

Basic schedule​

A basic schedule is defined by a JobDefinition and a cron_schedule using the ScheduleDefinition class. A job can be thought of as a selection of assets or operations executed together.

Simple Schedule Example
Loading...

Run schedules in a different timezone​

By default, schedules without a timezone will run in Coordinated Universal Time (UTC). If you want to run a schedule in a different timezone, you can set the timezone parameter.

daily_schedule = ScheduleDefinition(
    job=daily_refresh_job,
    cron_schedule="0 0 * * *",
    timezone="America/Los_Angeles",
)

Run schedules on a partitioned asset​

If you have a partitioned asset and job, you can create a schedule using the partition with build_schedule_from_partitioned_job. The schedule will execute as the same cadence specified by the partition definition.

Schedule with partition
Loading...

If you have a partitioned job, you can create a schedule from the partition using build_schedule_from_partitioned_job.

from dagster import build_schedule_from_partitioned_job, job


@job(config=partitioned_config)
def partitioned_op_job(): ...

partitioned_op_schedule = build_schedule_from_partitioned_job(
    partitioned_op_job,
)

Next steps​

By understanding and effectively using these automation methods, you can build more efficient data pipelines that respond to your specific needs and constraints.