使用Docker快速启动一个Spark集群

docker-compose 文件如下

version: '3'
services:
  master:
    image: bde2020/spark-master:2.3.1-hadoop2.7
    ports:
      - "8080:8080"
      - "7077:7077"
      - "6066:6066"
    environment:
      - INIT_DAEMON_STEP=setup_spark
  worker:
    image: bde2020/spark-worker:2.3.1-hadoop2.7
    depends_on:
      - master
    ports:
      - "8081"
    environment:
      - "SPARK_MASTER=spark://master:7077"

然后运行docker-compose up -d –scale worker=3　就可以启动一个有三个节点的集群了．