「CentOS Stream8」dockerでApache Sparkをインストールする方法

2021年3月26日

1.dockerバージョン確認
# docker –version
Docker version 20.10.5, build 55c4c88

2.docker-composeバージョン確認
# docker-compose –version
docker-compose version 1.28.5, build c4eb3a1f

3.「docker-spark.git」のダウンロード
# git clone https://github.com/Semantive/docker-spark.git
Cloning into 'docker-spark’…
remote: Enumerating objects: 26, done.
remote: Total 26 (delta 0), reused 0 (delta 0), pack-reused 26
Unpacking objects: 100% (26/26), 7.76 KiB | 378.00 KiB/s, done.

4.「docker-compose.yml」を確認
# cd docker-spark/
# cat docker-compose.yml

master:
  image: semantive/spark
  command: bin/spark-class org.apache.spark.deploy.master.Master -h master
  hostname: master
  environment:
    MASTER: spark://master:7077
    SPARK_CONF_DIR: /conf
    SPARK_PUBLIC_DNS: localhost
  ports:
    - 4040:4040
    - 6066:6066
    - 7077:7077
    - 8080:8080
  volumes:
    - ./data:/tmp/data

worker1:
  image: semantive/spark
  command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
  hostname: worker1
  environment:
    SPARK_CONF_DIR: /conf
    SPARK_WORKER_CORES: 4
    SPARK_WORKER_MEMORY: 2g
    SPARK_WORKER_PORT: 8881
    SPARK_WORKER_WEBUI_PORT: 8081
    SPARK_PUBLIC_DNS: localhost
  links:
    - master
  ports:
    - 8081:8081
  volumes:
    - ./data:/tmp/data

worker2:
  image: semantive/spark
  command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
  hostname: worker2
  environment:
    SPARK_CONF_DIR: /conf
    SPARK_WORKER_CORES: 4
    SPARK_WORKER_MEMORY: 2g
    SPARK_WORKER_PORT: 8882
    SPARK_WORKER_WEBUI_PORT: 8082
    SPARK_PUBLIC_DNS: localhost
  links:
    - master
  ports:
    - 8082:8082
  volumes:
    - ./data:/tmp/data
      

5.ファイアウォールの設定

# sudo firewall-cmd –add-port=8080/tcp –zone=public –permanent

success

# sudo firewall-cmd –reload

success

6.docker実行
# docker-compose up -d

Pulling master (semantive/spark:)...
latest: Pulling from semantive/spark
10a267c67f42: Pull complete
fb5937da9414: Pull complete
f131c9b3ecc4: Pull complete
f59cc3b26f35: Pull complete
63df868a0518: Pull complete
83282147e192: Pull complete
b9595cab9382: Pull complete
df2d57cc0364: Pull complete
2116ab3c81cc: Pull complete
471c9f9cd41d: Pull complete
79c19f8c71f4: Pull complete
d38cd9653d42: Pull complete
8042e0362cc0: Pull complete
Digest: sha256:cb093ec4febeeca8fe33a56e47f7af657d98c8af28dcd61ae8fea394de4af3f6
Status: Downloaded newer image for semantive/spark:latest
Creating docker-spark_master_1 ... done
Creating docker-spark_worker1_1 ... done
Creating docker-spark_worker2_1 ... done

7.動作確認
http://192.168.229.129:8080/