Dagster Cloud

Open Source vs Cloud

Dagster is Open Source software, meaning it is totally free to download, modify and run in your own environment.

Dagster Cloud is a cloud hosted version of Dagster which is managed by Elementl, the main commercial organisation that develops and supports Dagster.

The main value add of using a Managed Service such as Dagster Cloud is the avoidance of operational overhead. By using Dagster Cloud, you will not need to install, upgrade or monitor the platform and can leave this to the experts. This allows you to concentrate on the higher level activities specific to your own data domain.

Dagster Cloud also offers additional workflows for CI/CD, managing the process by which data changes are deployed.

A Tour Of Dagster Cloud

Visually, Dagster Cloud looks very similar to the Dagit user interface which by now you should be familiar with.

Deployment Models

Dagster Cloud has two deployment models:

• Serverless - In this model, Dagster Cloud will provide the compute that runs all of your jobs. This means that you do not have to setup or even concern your self with the notion of servers.
• Hybrid - In this model, Dagster Cloud will manage the Dagster software, but use compute servers hosted in your own data centre or cloud account.

Serverless is the simplest option, but your business might be attracted to Hybrid for the following reasons:

• Data Security and compliance - where you need to keep data processing on premise or in your own account for legal reasons, perhaps related to data sovereignty or your businesses information security rules;
• Cost - Perhaps it is more cost efficient for you to use your own servers, for instance where you have pre-existing contracts or spare infrastructure;
• System Integration - Perhaps your data integrates with other systens or databases which you cannot expose over the internet.

These types of situations are common in the data engineering world, which makes the Hybrid deployment option very appealing. It does however require slightly more integration and engineering work to get up and running.

Dagster Cloud CLI

Dagster Cloud can be administed using a CLI tool. This can be installed from pip.

pip install dagster-cloud