Spaces:
Paused
Paused
| --- | |
| title: Docker Compose | |
| hide_title: true | |
| sidebar_position: 5 | |
| version: 1 | |
| --- | |
| import useBaseUrl from '@docusaurus/useBaseUrl'; | |
| # Using Docker Compose | |
| <img src={useBaseUrl('/img/docker-compose.webp')} width="150" /> | |
| <br /> | |
| <br /> | |
| Since `docker compose` is primarily designed to run a set of containers on **a single host** | |
| and can't support requirements for **high availability**, we do not support nor recommend | |
| using our `docker compose` constructs to support production-type use-cases. For single host | |
| environments, we recommend using [minikube](https://minikube.sigs.k8s.io/docs/start/) along | |
| with our [installing on k8s](https://superset.apache.org/admin-docs/installation/running-on-kubernetes) | |
| documentation. | |
| As mentioned in our [quickstart guide](/user-docs/quickstart), the fastest way to try | |
| Superset locally is using Docker Compose on a Linux or Mac OSX | |
| computer. Superset does not have official support for Windows. It's also the easiest | |
| way to launch a fully functioning **development environment** quickly. | |
| Note that there are 4 major ways we support to run `docker compose`: | |
| 1. **docker-compose.yml:** for interactive development, where we mount your local folder with the | |
| frontend/backend files that you can edit and experience the changes you | |
| make in the app in real time | |
| 1. **docker-compose-light.yml:** a lightweight configuration with minimal services (database, | |
| Superset app, and frontend dev server) for development. Uses in-memory caching instead of Redis | |
| and is designed for running multiple instances simultaneously | |
| 1. **docker-compose-non-dev.yml** where we just build a more immutable image based on the | |
| local branch and get all the required images running. Changes in the local branch | |
| at the time you fire this up will be reflected, but changes to the code | |
| while `up` won't be reflected in the app | |
| 1. **docker-compose-image-tag.yml** where we fetch an image from docker-hub say for the | |
| `5.0.0` release for instance, and fire it up so you can try it. Here what's in | |
| the local branch has no effects on what's running, we just fetch and run | |
| pre-built images from docker-hub. For `docker compose` to work along with the | |
| Postgres image it boots up, you'll want to point to a `-dev`-suffixed TAG, as in | |
| `export TAG=5.0.0-dev` or `export TAG=4.1.2-dev`, with `latest-dev` being the default. | |
| The `dev` builds include the `psycopg2-binary` required to connect | |
| to the Postgres database launched as part of the `docker compose` builds. | |
| More on these approaches after setting up the requirements for either. | |
| ## Requirements | |
| Note that this documentation assumes that you have [Docker](https://www.docker.com) and | |
| [git](https://git-scm.com/) installed. Note also that we used to use `docker-compose` but that | |
| is on the path to deprecation so we now use `docker compose` instead. | |
| ## 1. Clone Superset's GitHub repository | |
| [Clone Superset's repo](https://github.com/apache/superset) in your terminal with the | |
| following command: | |
| ```bash | |
| git clone --depth=1 https://github.com/apache/superset.git | |
| ``` | |
| Once that command completes successfully, you should see a new `superset` folder in your | |
| current directory. | |
| ## 2. Launch Superset Through Docker Compose | |
| First let's assume you're familiar with `docker compose` mechanics. Here we'll refer generally | |
| to `docker compose up` even though in some cases you may want to force a check for newer remote | |
| images using `docker compose pull`, force a build with `docker compose build` or force a build | |
| on latest base images using `docker compose build --pull`. In most cases though, the simple | |
| `up` command should do just fine. Refer to docker compose docs for more information on the topic. | |
| ### Option #1 - for an interactive development environment | |
| ```bash | |
| # The --build argument insures all the layers are up-to-date | |
| docker compose up --build | |
| ``` | |
| When running in development mode the `superset-node` | |
| container needs to finish building assets in order for the UI to render properly. If you would just | |
| like to try out Superset without making any code changes follow the steps documented for | |
| `production` or a specific version below. | |
| By default, we mount the local superset-frontend folder here and run `npm install` as well | |
| as `npm run dev` which triggers webpack to compile/bundle the frontend code. Depending | |
| on your local setup, especially if you have less than 16GB of memory, it may be very slow to | |
| perform those operations. In this case, we recommend you set the env var | |
| `BUILD_SUPERSET_FRONTEND_IN_DOCKER` to `false`, and to run this locally instead in a terminal. | |
| Simply trigger `npm i && npm run dev`, this should be MUCH faster. | |
| Sometimes, your npm-related state can get out-of-wack, running `npm run prune` from | |
| the `superset-frontend/` folder will nuke the various' packages `node_module/` folders | |
| and help you start fresh. In the context of `docker compose` setting | |
| `export NPM_RUN_PRUNE=true` prior to running `docker compose up` will trigger that | |
| from within docker. This will slow down the startup, but will fix various npm-related issues. | |
| ### Option #2 - lightweight development with multiple instances | |
| For a lighter development setup that uses fewer resources and supports running multiple instances: | |
| ```bash | |
| # Single lightweight instance (default port 9001) | |
| docker compose -f docker-compose-light.yml up | |
| # Multiple instances with different ports | |
| NODE_PORT=9001 docker compose -p superset-1 -f docker-compose-light.yml up | |
| NODE_PORT=9002 docker compose -p superset-2 -f docker-compose-light.yml up | |
| NODE_PORT=9003 docker compose -p superset-3 -f docker-compose-light.yml up | |
| ``` | |
| This configuration includes: | |
| - PostgreSQL database (internal network only) | |
| - Superset application server | |
| - Frontend development server with webpack hot reloading | |
| - In-memory caching (no Redis) | |
| - Isolated volumes and networks per instance | |
| Access each instance at `http://localhost:{NODE_PORT}` (e.g., `http://localhost:9001`). | |
| ### Option #3 - build a set of immutable images from the local branch | |
| ```bash | |
| docker compose -f docker-compose-non-dev.yml up | |
| ``` | |
| ### Option #4 - boot up an official release | |
| ```bash | |
| # Set the version you want to run | |
| export TAG=5.0.0 | |
| # Fetch the tag you're about to check out (assuming you shallow-cloned the repo) | |
| git fetch --depth=1 origin tag $TAG | |
| # Could also fetch all tags too if you've got bandwidth to spare | |
| # git fetch --tags | |
| # Checkout the corresponding git ref | |
| git checkout $TAG | |
| # Fire up docker compose | |
| docker compose -f docker-compose-image-tag.yml up | |
| ``` | |
| Here various release tags, github SHA, and latest `master` can be referenced by the TAG env var. | |
| Refer to the docker-related documentation to learn more about existing tags you can point to | |
| from Docker Hub. | |
| For option #2 and #3, we recommend checking out the release tag from the git repository | |
| (ie: `git checkout 5.0.0`) for more guaranteed results. This ensures that the `docker-compose.*.yml` | |
| configurations and that the mounted `docker/` scripts are in sync with the image you are | |
| looking to fire up. | |
| ## `docker compose` tips & configuration | |
| All of the content belonging to a Superset instance - charts, dashboards, users, etc. - is stored in | |
| its metadata database. In production, this database should be backed up. The default installation | |
| with docker compose will store that data in a PostgreSQL database contained in a Docker | |
| [volume](https://docs.docker.com/storage/volumes/), which is not backed up. | |
| Again, **THE DOCKER-COMPOSE INSTALLATION IS NOT PRODUCTION-READY OUT OF THE BOX.** | |
| You should see a stream of logging output from the containers being launched on your machine. Once | |
| this output slows, you should have a running instance of Superset on your local machine! To avoid | |
| the wall of text on future runs, add the `-d` option to the end of the `docker compose up` command. | |
| ### Configuring Further | |
| The following is for users who want to configure how Superset runs in Docker Compose; otherwise, you | |
| can skip to the next section. | |
| You can install additional python packages and apply config overrides by following the steps | |
| mentioned in [docker/README.md](https://github.com/apache/superset/tree/master/docker#configuration) | |
| Note that `docker/.env` sets the default environment variables for all the docker images | |
| used by `docker compose`, and that `docker/.env-local` can be used to override those defaults. | |
| Also note that `docker/.env-local` is referenced in our `.gitignore`, | |
| preventing developers from risking committing potentially sensitive configuration to the repository. | |
| One important variable is `SUPERSET_LOAD_EXAMPLES` which determines whether the `superset_init` | |
| container will populate example data and visualizations into the metadata database. These examples | |
| are helpful for learning and testing out Superset but unnecessary for experienced users and | |
| production deployments. The loading process can sometimes take a few minutes and a good amount of | |
| CPU, so you may want to disable it on a resource-constrained device. | |
| For more advanced or dynamic configurations that are typically managed in a `superset_config.py` file | |
| located in your `PYTHONPATH`, note that it can be done by providing a | |
| `docker/pythonpath_dev/superset_config_docker.py` that will be ignored by git | |
| (preventing you to commit/push your local configuration back to the repository). | |
| The mechanics of this are in `docker/pythonpath_dev/superset_config.py` where you can see | |
| that the logic runs a `from superset_config_docker import *` | |
| Users often want to connect to other databases from Superset. Currently, the easiest way to | |
| do this is to modify the `docker-compose-non-dev.yml` file and add your database as a service that | |
| the other services depend on (via `x-superset-depends-on`). Others have attempted to set | |
| `network_mode: host` on the Superset services, but these generally break the installation, | |
| because the configuration requires use of the Docker Compose DNS resolver for the service names. | |
| If you have a good solution for this, let us know! | |
| Superset uses [Scarf Gateway](https://about.scarf.sh/scarf-gateway) to collect telemetry | |
| data. Knowing the installation counts for different Superset versions informs the project's | |
| decisions about patching and long-term support. Scarf purges personally identifiable information | |
| (PII) and provides only aggregated statistics. | |
| To opt-out of this data collection for packages downloaded through the Scarf Gateway by your docker | |
| compose based installation, edit the `x-superset-image:` line in your `docker-compose.yml` and | |
| `docker-compose-non-dev.yml` files, replacing `apachesuperset.docker.scarf.sh/apache/superset` with | |
| `apache/superset` to pull the image directly from Docker Hub. | |
| To disable the Scarf telemetry pixel, set the `SCARF_ANALYTICS` environment variable to `false` in | |
| your `docker/.env` file. This is read at runtime, so it disables the pixel on the pre-built image | |
| without rebuilding the frontend. | |
| ## 3. Log in to Superset | |
| Your local Superset instance also includes a Postgres server to store your data and is already | |
| pre-loaded with some example datasets that ship with Superset. You can access Superset now via your | |
| web browser by visiting `http://localhost:8088`. Note that many browsers now default to `https` - if | |
| yours is one of them, please make sure it uses `http`. | |
| Log in with the default username and password: | |
| ```bash | |
| username: admin | |
| ``` | |
| ```bash | |
| password: admin | |
| ``` | |
| ## 4. Connecting Superset to your local database instance | |
| When running Superset using `docker` or `docker compose` it runs in its own docker container, as if | |
| the Superset was running in a separate machine entirely. Therefore attempts to connect to your local | |
| database with the hostname `localhost` won't work as `localhost` refers to the docker container | |
| Superset is running in, and not your actual host machine. Fortunately, docker provides an easy way | |
| to access network resources in the host machine from inside a container, and we will leverage this | |
| capability to connect to our local database instance. | |
| Here the instructions are for connecting to postgresql (which is running on your host machine) from | |
| Superset (which is running in its docker container). Other databases may have slightly different | |
| configurations but gist would be same and boils down to 2 steps - | |
| 1. **(Mac users may skip this step)** Configuring the local postgresql/database instance to accept | |
| public incoming connections. By default, postgresql only allows incoming connections from | |
| `localhost` and under Docker, unless you use `--network=host`, `localhost` will refer to different | |
| endpoints on the host machine and in a docker container respectively. Allowing postgresql to accept | |
| connections from the Docker involves making one-line changes to the files `postgresql.conf` and | |
| `pg_hba.conf`; you can find helpful links tailored to your OS / PG version on the web easily for | |
| this task. For Docker it suffices to only whitelist IPs `172.0.0.0/8` instead of `*`, but in any | |
| case you are _warned_ that doing this in a production database _may_ have disastrous consequences as | |
| you are opening your database to the public internet. | |
| 1. Instead of `localhost`, try using `host.docker.internal` (Mac users, Ubuntu) or `172.18.0.1` | |
| (Linux users) as the hostname when attempting to connect to the database. This is a Docker internal | |
| detail -- what is happening is that, in Mac systems, Docker Desktop creates a dns entry for the | |
| hostname `host.docker.internal` which resolves to the correct address for the host machine, whereas | |
| in Linux this is not the case (at least by default). If neither of these 2 hostnames work then you | |
| may want to find the exact hostname you want to use, for that you can do `ifconfig` or | |
| `ip addr show` and look at the IP address of `docker0` interface that must have been created by | |
| Docker for you. Alternately if you don't even see the `docker0` interface try (if needed with sudo) | |
| `docker network inspect bridge` and see if there is an entry for `"Gateway"` and note the IP | |
| address. | |
| ## 4. To build or not to build | |
| When running `docker compose up`, docker will build what is required behind the scene, but | |
| may use the docker cache if assets already exist. Running `docker compose build` prior to | |
| `docker compose up` or the equivalent shortcut `docker compose up --build` ensures that your | |
| docker images match the definition in the repository. This should only apply to the main | |
| docker-compose.yml file (default) and not to the alternative methods defined above. | |