How to install and use Docker in Ubuntu

Containerization platform, Container Engine

Work Instruction

Install Docker

Check docker is already installed or not

docker --version

if not installed, install it

sudo apt update # update package manager
sudo apt install apt-transport-https ca-certificates curl software-properties-common # install required package
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg # Add official GPG key of docker
echo "deb [signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null # Add docker repository
sudo apt update # update package manager again
sudo apt install docker-ce # install docker
sudo systemctl start docker # start docker
sudo systemctl enable docker # set docker start when bootup

Practice: run pytorch image with nvidia toolkit with volume mount between host and container

Install Nvidia container toolkit

# add nvidia repository (add GPG key)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update # update packages
sudo apt-get install -y nvidia-container-toolkit # install nvidia container toolkit
sudo nvidia-ctk runtime configure --runtime=docker # config toolkit runtime to support toolkit and docker 
sudo systemctl restart docker # restart to use new configuration
sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi # execute docker container using nvidia container toolkit
# sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 nvidia-smi

Pull Nvidia Docker image which is including Pytorch

docker pull nvcr.io/nvidia/pytorch:23.04-py3

Mount volume between host and container
- Go to root and run terminal
- Generate directory in host at root

mkdir ML_Host_Dir

Run docker container

docker run -it -v /ML_Host_Dir/:/ML_Container_Dir/ --ipc=host --gpus all --name ML_container_Pytorch nvcr.io/nvidia/pytorch:23.04-py3

options
- -it: interatice and assign virtual terminal
- -v /ML_Host_Dir/:/ML_Container_Dir/: volume mount between host and container
- –ipc=host: container use namespace of host
- –gpus all: use all gpus
- –name ML_container_Pytorch: assign name of the image
- nvcr.io/nvidia/pytorch:23.04-py3: image to run
Check pytorch is running well in terminal in container

python3

import torch
torch.__version__

Check pytorch and cuda is available

torch.cuda.is_availble()

Go to ML_Host_Dir generate folder test

mkdir test

Go to vitual environment terminal (container interface) and check test dir is generated

cd ML_Container_Dir
ls

Generate example.py

sudo touch example.py
sudo chmod 777 example.py # give full access

example.py

import numpy as np
print(np.log10(2))

Go to container terminal and run it

python3 example.py

Docker image가 .tar인 경우

docker load -i /path/to/yourimage.tar # load
docker tag [Old Image Name]:[Old Image Tag] [New Image Name]:[New Image Tag] # rename tag
docker rmi [Old Image Name]:[Old Image Tag] # remove original image. Untagged

Commands

sudo systemctl status docker # Check docker daemon is active
sudo docker ps # list docker containers which is current running
sudo docker info # docker deamon status and info
sudo docker images # check images list
sudo docker build -f Dockerfile -t fun-docker . # build image
sudo docker run -d -p 8080:8080 fun-docker # run image -p host port number : container port number if web application, image name
sudo docker ps # check running image
sudo docker logs 392acbcbdaa0 # check log (image id)
sudo docker login # login to dockerhub
sudo docker tag fun-docker:latest koyumi0601/docker-example-dreamcoding:latest # change tag_name
sudo docker push koyumi0601/docker-example-dreamcoding:latest # account_name/repository_name, push dockerhub
sudo docker stop 392acbcbdaa0 # container name or container id
sudo docker start 392acbcbdaa0 # start docker container again
sudo docker attach 392acbcbdaa0 # terminal

sudo docker rm 392acbcbdaa0 # remove in runnable list (after stop)
sudo docker rmi 392acbcbdaa0 # remove container
sudo apt purge docker-ce docker-ce-cli containerd.io # remove docker
sudo rm -rf /var/lib/docker # remove docker configuration. 
# need to backup docker related configuration before remove configuration

check nvidia info

sudo nvidia-smi # check nvidia info

Generate Docker image

dockerfile

.tar

Practice: Generate docker image using dockerfile and push to dockerhub (from dreamcoding)

Get example source code from github

https://github.com/dream-ellie/docker-example
dockerfile

# Order by Layer0 -> Layer4
# base image
FROM node:16-alpine

# wokring directory
WORKDIR /app

# app dependencies
COPY package.json package-lock.json ./
RUN npm ci

# copy source code
COPY index.js .

# execute app command
ENTRYPOINT [ "node", "index.js" ]

Build image

docker build -f Dockerfile -t fun-docker .
# -f: target dockerfile name to build 
# -t: docker image name, so-called tag
# .: required dockerfile path, so-called build context

2023-08-07_11-01-docker-5

Check images

docker images

2023-08-07_11-01-docker-6

Run image

docker run -d -p 8080:8080 fun-docker
# -d: detached terminal.
# -p: port connect between host(8080) and container(8080)

2023-08-07_11-01-docker-7

Check current running docker image

docker ps

2023-08-07_11-01-docker-8

Go to localhost:8080 and check application is running well

2023-08-07_11-01-docker-9

Check log

docker logs 392acbcbdaa0

Upload image to container registry (dockerhub)

Go to https://hub.docker.com/
Create repository: koyumi0601/docker-example-dreamcoding:latest
Log-in(terminal)

docker login

Change docker tag to dockerhub repository name

docker tag fun-docker:latest koyumi0601/docker-example-dreamcoding:latest # change tag_name

push

docker push koyumi0601/docker-example-dreamcoding:latest # account_name/repository_name

Result

2023-08-07_11-01-docker-10

Practice: teddy note

Pull docker image from dockerhub

Go to tensorflow/tensorflow https://hub.docker.com/r/tensorflow/tensorflow/tags (example)
Click gray area to copy command to clipboard

2023-08-05_10-42-docker

Go to project folder
pull docker image in terminal

docker pull tensorflow/tensorflow:2.13.0-gpu-jupyter # Tensorflow
# docker pull gcr.io/kaggle-gpu-images/python:latest
docker pull nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 # NVIDIA GPU

Edit dockerfile

Get dockerfile from github repository
- https://github.com/teddylee777/deepko
Base image

# base images
FROM tensorflow/tensorflow:2.13.0-gpu-jupyter
FROM nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 AS nvidia

CUDA settings
- Refer to kaggle python: https://github.com/Kaggle/docker-python but not changed

# CUDA
ENV CUDA_MAJOR_VERSION=11
ENV CUDA_MINOR_VERSION=8
ENV CUDA_VERSION=$CUDA_MAJOR_VERSION.$CUDA_MINOR_VERSION

ENV PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/bin:${PATH}

ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
ENV LD_LIBRARY_PATH_NO_STUBS="/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/opt/conda/lib"
ENV LD_LIBRARY_PATH="/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:/opt/conda/lib"
ENV NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility
ENV NVIDIA_REQUIRE_CUDA="cuda>=$CUDA_MAJOR_VERSION.$CUDA_MINOR_VERSION"

ENV TZ=Asia/Seoul
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

Mirror server

# 카카오 ubuntu archive mirror server 추가. 다운로드 속도 향상
RUN sed -i 's@archive.ubuntu.com@mirror.kakao.com@g' /etc/apt/sources.list && \
    apt-get update && apt-get install alien -y

Applications

# openjdk java vm 설치
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    wget \
    build-essential \
    libboost-dev \
    libboost-system-dev \
    libboost-filesystem-dev \
    g++ \
    gcc \
    openjdk-8-jdk \
    python3-dev \
    python3-pip \
    curl \
    bzip2 \
    ca-certificates \
    libglib2.0-0 \
    libxext6 \
    libsm6 \
    libxrender1 \
    libssl-dev \
    libzmq3-dev \
    vim \
    git \
    cmake

RUN apt-get update

ARG CONDA_DIR=/opt/conda
ENV CUDA_PATH=/usr/local/cuda

# add to path
ENV PATH $CONDA_DIR/bin:$PATH

# Install miniconda
RUN echo "export PATH=$CONDA_DIR/bin:"'$PATH' > /etc/profile.d/conda.sh && \
    curl -sL https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o ~/miniconda.sh && \
    /bin/bash ~/miniconda.sh -b -p $CONDA_DIR && \
    rm ~/miniconda.sh

# Conda 가상환경 생성
RUN conda config --set always_yes yes --set changeps1 no && \
    conda create -y -q -n py39 python=3.9

ENV PATH /opt/conda/envs/py39/bin:$PATH
ENV CONDA_DEFAULT_ENV py39
ENV CONDA_PREFIX /opt/conda/envs/py39

RUN apt-get update

# Install OpenCL & libboost
RUN apt-get install -y ocl-icd-libopencl1 clinfo libboost-all-dev && \
    mkdir -p /etc/OpenCL/vendors && \
    echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd

Python packages

# 패키지 설치
RUN pip install setuptools && \
    pip install mkl && \
    pip install pymysql && \
    pip install numpy && \
    pip install scipy && \
    pip install pandas==1.5.3 && \
    pip install matplotlib && \
    pip install seaborn && \
    pip install hyperopt && \
    pip install optuna && \
    pip install missingno && \
    pip install mlxtend && \
    pip install catboost && \
    pip install kaggle && \
    pip install folium && \
    pip install librosa && \
    pip install nbconvert && \
    pip install Pillow && \
    pip install tqdm && \
    pip install tensorflow-datasets && \
    pip install gensim && \
    pip install nltk && \
    pip install wordcloud && \
    pip install statsmodels

RUN apt-get update

RUN apt-get install -y graphviz && \
    apt-get install -y graphviz-dev && \
    pip install pygraphviz

RUN pip install graphviz && \
    pip install cupy-cuda11x

RUN pip install --upgrade cython && \
    pip install --upgrade cysignals && \
    pip install pyfasttext && \
    pip install fasttext && \
    pip install accelerate && \
    pip install sentencepiece && \
    pip install -U bitsandbytes && \
    pip install -U git+https://github.com/huggingface/transformers.git && \
    pip install -U git+https://github.com/huggingface/peft.git && \
    pip install -U git+https://github.com/huggingface/accelerate.git && \
    pip install -U datasets

RUN pip install pystan==2.19.1.1 && \
    pip install prophet && \
    pip install torchsummary

RUN pip install wandb tensorboard albumentations pydicom opencv-python scikit-image pyarrow kornia \
    catalyst captum

RUN pip install fastai && \
    pip install fvcore 

RUN pip install openai && \
    pip install langchain && \
    pip install duckduckgo-search && \
    pip install pypdf

RUN pip install pandas==1.5.3

RUN pip install cudf-cu11 dask-cudf-cu11 cuml-cu11 --extra-index-url=https://pypi.nvidia.com

RUN pip install -U "ipython[all]"

RUN git clone https://github.com/slundberg/shap.git && cd shap && \
    python setup.py install

# Jupyter Notebook Extension 설정
# ARG CONDA_DIR=/opt/conda/envs/py39

#RUN conda install -c conda-forge jupyter && \
#    conda install -c conda-forge jupyterlab && \
#    conda install -c conda-forge notebook && \
#    conda install -c conda-forge jupyter_nbextensions_configurator && \
#    conda install -c conda-forge jupyter_contrib_nbextensions && \
#    conda install -c conda-forge jupyterthemes

#RUN conda install -c conda-forge ipywidgets
RUN pip install jupyter jupyterlab notebook jupyterthemes ipywidgets
# RUN jupyter nbextensions_configurator enable
# RUN jupyter contrib nbextension install
# RUN jupyter nbextension enable --py widgetsnbextension --sys-prefix
 
ENV PATH=/usr/local/bin:${PATH}

# 나눔고딕 폰트 설치
# matplotlib에 Nanum 폰트 추가
RUN apt-get update && apt-get install fonts-nanum*
RUN cp /usr/share/fonts/truetype/nanum/Nanum* /opt/conda/envs/py39/lib/python3.9/site-packages/matplotlib/mpl-data/fonts/ttf/ && \
    fc-cache -fv && \
    rm -rf ~/.cache/matplotlib/*

# konlpy, py-hanspell, soynlp 패키지 설치 
RUN pip install konlpy

# 형태소 분석기 mecab 설치
# RUN cd /tmp && \
#    wget "https://www.dropbox.com/s/9xls0tgtf3edgns/mecab-0.996-ko-0.9.2.tar.gz?dl=1" && \
#    tar zxfv mecab-0.996-ko-0.9.2.tar.gz?dl=1 && \
#    cd mecab-0.996-ko-0.9.2 && \
#    ./configure && \
#    make && \
#    make check && \
#    make install && \
#    ldconfig

# RUN apt-get update

# RUN cd /tmp && \
#   wget "https://www.dropbox.com/s/i8girnk5p80076c/mecab-ko-dic-2.1.1-20180720.tar.gz?dl=1" && \
#    apt install -y autoconf && \
#    tar zxfv mecab-ko-dic-2.1.1-20180720.tar.gz?dl=1 && \
#    cd mecab-ko-dic-2.1.1-20180720 && \
#    ./autogen.sh && \
#    ./configure && \
#    make && \
#    make install && \
#    ldconfig

# 형태소 분석기 mecab 파이썬 패키지 설치
# RUN cd /tmp && \
#    git clone https://bitbucket.org/eunjeon/mecab-python-0.996.git && \
#    cd mecab-python-0.996 && \
#    python setup.py build && \
#    python setup.py install

# RUN apt-get update

# XGBoost (GPU 설치)
RUN pip install xgboost && \
    pip install lightgbm

#RUN conda install -c nvidia -y

# Install OpenCL & libboost (required by LightGBM GPU version)
# RUN apt-get install -y ocl-icd-libopencl1 clinfo libboost-all-dev && \
#    mkdir -p /etc/OpenCL/vendors && \
#    echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd

# RUN conda install -c conda-forge py-xgboost-gpu -y

# lightgbm (GPU 설치)
#RUN pip uninstall -y lightgbm && \
#    cd /usr/local/src && mkdir lightgbm && cd lightgbm && \
#    git clone --recursive --branch stable --depth 1 https://github.com/microsoft/LightGBM && \
#    cd LightGBM && mkdir build && cd build && \
#    cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ .. && \
#    make -j$(nproc) OPENCL_HEADERS=/usr/local/cuda-11.2/targets/x86_64-linux/include LIBOPENCL=/usr/local/cuda-11.2/targets/x86_64-linux/lib && \
#    cd /usr/local/src/lightgbm/LightGBM/python-package && python setup.py install --precompile


# RUN git clone --recursive https://github.com/microsoft/LightGBM && \
#    cd LightGBM && \
#    mkdir build && \
#    cd build && \
#    cmake -DUSE_GPU=1 .. && \
#    make -j$(nproc) && \
#    cd ..

# soynlp, KR-WordRank, soyspacing, customized_konlpy 설치
RUN pip install soynlp && \
    pip install krwordrank && \
    pip install soyspacing && \
    pip install customized_konlpy

# PyTorch 2.0 설치
RUN pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
RUN pip install torchtext==0.15.1

# PyTorch 2.0 compile 모드 지원을 위한 설치
RUN pip install torchtriton --extra-index-url "https://download.pytorch.org/whl/nightly/cu118" 

# TensorFlow 2.13.0 설치
RUN pip install tensorflow==2.13.0

# Remove the CUDA stubs.
ENV LD_LIBRARY_PATH="$LD_LIBRARY_PATH_NO_STUBS"

# locale 설정
# RUN apt-get update && apt-get install -y locales tzdata && \
#     locale-gen ko_KR.UTF-8 && locale -a && \
#     ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime

RUN apt-get autoremove -y && apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    conda clean -a -y

RUN apt-get update

# LANG 환경변수 설정
ENV LANG ko_KR.UTF-8

# Jupyter Notebook config 파일 생성
# RUN jupyter notebook --generate-config

# config 파일 복사 (jupyter_notebook_config.py 파일 참고)
# COPY jupyter_notebook_config.py /root/.jupyter/jupyter_notebook_config.py

# 설치 완료 후 테스트용 ipynb
COPY ./01-GPU-TEST/GPU-Test.ipynb /home/jupyter/GPU-Test.ipynb

# 기본
EXPOSE 8888
# jupyter notebook 의 password를 지정하지 않으면 보안상 취약하므로 지정하는 것을 권장
# CMD jupyter notebook --allow-root

Commands

docker images # check images list
docker build -f gpu.Dockerfile -t koyumi0601/docker-example-teddy:latest . # build image
# docker run -d -p 8080:8080 fun-docker # run image
docker run --runtime=nvidia --rm -itd -p 8888:8888 -v /data/jupyter_data:/home/jupyter koyumi0601/docker-example-teddy:latest jupyter notebook --allow-root --ip=0.0.0.0 --port=8888 --no-browser --NotebookApp.password= --notebook-dir=/home/jupyter
# docker run # command to run
# --runtime=nvidia # use GPU
# --rm # remove container after run
# -itd # interactice, detached mode, background. user can interact terminal
# -p 8888:8888 # connect host port 8888 and container port 8888
# -v /data/jupyter_data:/home/jupyter # mount volume to share files between host and container. it requires directory in host before run
# jupyter notebook --allow-root --ip=0.0.0.0 --port=8888 --no-browser --NotebookApp.password= # command to run jupyter notebook
# --allow-root # allow user as root user to run jupyter notebook
# --ip=0.0.0.0 # allow all ip address
# --port=8888 # set jupyter notebook port as 8888
# --no-browser # not open browser automatically
# --NotebookApp.password= # run without password
# now, jupyter notebook is running in background. to see it, open browser
# http://127.0.0.1:8888

Insert token

2023-08-07_21-43-docker-token

35ee2047023b130b847b727347425b2bc3523062ef5d770e

2023-08-07_21-43-docker-11

GPU-test-도커

docker ps # check running image
docker logs cc1a7386ec38 # check log (image id)
docker login # login to dockerhub
# docker tag fun-docker:latest koyumi0601/docker-example-dreamcoding:latest # change tag_name
docker push koyumi0601/docker-example-teddy:latest # account_name/repository_name, push dockerhub
docker stop cc1a7386ec38 # container name or container id
docker start cc1a7386ec38 # start docker container again
docker rm cc1a7386ec38 # remove docker after stop

Reference

Blog https://89douner.tistory.com/96
테디노트 영상 https://www.youtube.com/watch?v=Tw7dU-9AkmU&t=305s
드림코딩 영상 https://www.youtube.com/watch?v=LXJhA3VWXFA
Docker docs https://docs.docker.com/
TensorFlow Docker Images (recommended) https://hub.docker.com/r/tensorflow/tensorflow/
Deeplearning Docker Image, Deepo https://hub.docker.com/r/ufoym/deepo/
Kaggle/docker-python image https://github.com/Kaggle/docker-python

Share on

Twitter Facebook LinkedIn

How to install and use Docker in Ubuntu

Work Instruction

Install Docker

Practice: run pytorch image with nvidia toolkit with volume mount between host and container

Docker image가 .tar인 경우

Commands

Generate Docker image

dockerfile

.tar

Practice: Generate docker image using dockerfile and push to dockerhub (from dreamcoding)

Get example source code from github

Build image

Check images

Run image

Check current running docker image

Go to localhost:8080 and check application is running well

Check log

Upload image to container registry (dockerhub)

Practice: teddy note

Pull docker image from dockerhub

Edit dockerfile

Commands

Insert token

GPU-test-도커

Reference

Share on

You may also enjoy

CUDA example code 1. main.cpp, kernel.cu, kernel.cuh 구성으로 빌드하기, ubuntu

CMake

Tesseract OCR and OpenCV in Ubuntu

Qt for Linux