OVS, Docker, CentOS7 환경 만들기

Blog Logo
on



read

Vagrant

centos7기반의 ovs와 docker가 머신이 당분간 계속 필요할 것 같아서, vagrant를 이용하여 VM을 빌드해보기로 하였습니다. vagrant는 가상 머신 설치를 도와주는 프로그램입니다. vagrant가 없었던 시절에는 VM에 직접 인스톨 한 후 VM 이미지를 복사해 두어 재 사용하는 방법을 사용했었습니다. vagrant에서는 기본 이미지 (box라고 합니다.)를 제공하면서, 그 위에 설치 프로그램 형상을 사용자가 지정할 수 있도록 합니다. 쓰면 매우 편한겁니다.



1
2
3



$ uname -a
Linux cwyang 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ sudo apt-get install virtualbox vagrant

SSL 접속환경 확인

다음 ssl domain들이 인증서 피닝을 요구합니다.

  • vagrantcloud.com
  • cbs.centos.org
  • yum.dockerproject.org
  • raw.githubusercontent.com

회사안에서 SSL 복호화 장비가 있는 경우 회사 인증서를 VM에 신뢰할 수 있는 인증서로 심거나, SSL 복호화 장비에서 위의 사이트들에 대하여 복호화를 바이패스 하도록 설정합니다.

설치

github에 설치 스크립트를 작업해 둔 분이 있습니다. (링크)그 파일들을 이용해서 설치해 보았습니다. 감사합니다.

아래 두 파일을 저장한 후 vagrant up 명령어를 실행하면 수 분 안에 centos/7을 다운 받고 dependency package를 설치하고 ovs를 설치하고 docker를 설치해줍니다. 좋은 세상입니다.

설치 확인



1
2
3



$ vagrant ssh
Last login: Fri Oct 12 01:13:18 2018 from 10.0.2.2
[vagrant@localhost ~]$

호스트의 id_rsa.pub를 VM의 .ssh/authorized_keys에 등록하면 직접 ssh도 가능합니다.



1
2
3



$ ssh vagrant@127.0.0.1 -p 2222
Last login: Fri Oct 12 01:22:25 2018 from 10.0.2.2
[vagrant@localhost ~]$

도커와 OvS를 확인합니다.



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19



[vagrant@localhost ~]$ systemctl status openvswitch docker
● openvswitch.service - Open vSwitch
   Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2018-10-12 01:05:18 UTC; 21min ago
 Main PID: 25616 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
   CGroup: /system.slice/openvswitch.service

● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-10-12 01:05:37 UTC; 21min ago
     Docs: https://docs.docker.com
 Main PID: 25686 (dockerd)
    Tasks: 16
   Memory: 16.0M
   CGroup: /system.slice/docker.service
           ├─25686 /usr/bin/dockerd
           └─25689 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 -...

도커 실행을 위해서 docker그룹에 vagrant 사용자를 등록합니다.



1



[vagrant@localhost ~]$ sudo usermod -aG docker vagrant

Two containers w/ OVS bridge

구성



1
2
3



moby1: eth1 192.168.0.1/24
  | (ovs-br1)
moby2: eth1 192.168.0.2/24

OVS 브릿지 생성



1
2
3
4
5
6
7
8



root@localhost vagrant]# ovs-vsctl add-br ovs-br1
root@localhost vagrant]# ovs-vsctl show
94185d0-937e-49bf-a0a4-69ca5027718b
   Bridge "ovs-br1"
       Port "ovs-br1"
           Interface "ovs-br1"
               type: internal
   ovs_version: "2.9.0"

컨테이너 생성후 ovs-bridge에 연결



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19



[vagrant@localhost ~]$ docker run -dit --name=moby1 ubuntu
[vagrant@localhost ~]$ docker exec -it moby1 apt-get update
[vagrant@localhost ~]$ docker exec -it moby1 apt-get install iproute2 iputils-ping
[vagrant@localhost ~]$ sudo ovs-docker add-port ovs-br1 eth1 moby1 --ipaddress=192.168.0.1/24

[vagrant@localhost ~]$ docker run -dit --name=moby2 ubuntu
[vagrant@localhost ~]$ docker exec -it moby2 apt-get update
[vagrant@localhost ~]$ docker exec -it moby2 apt-get install iproute2 iputils-ping
[vagrant@localhost ~]$ sudo ovs-docker add-port ovs-br1 eth2 moby1 --ipaddress=192.168.0.2/24

[vagrant@localhost ~]$ docker exec -it moby1 ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
...
21: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
...
23: eth1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:bf:61:25:7e:f8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.1/24 scope global eth1
       valid_lft forever preferred_lft forever

컨테이너간 연결 확인



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17



[vagrant@localhost ~]$ sudo ovs-vsctl list-ports ovs-br1
6b9045d2be344_l
90cd4018819e4_l
[vagrant@localhost ~]$ docker exec moby1 ping 192.168.0.2 -c 1
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=0.037 ms

--- 192.168.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.037/0.037/0.037/0.000 ms
[vagrant@localhost ~]$ docker exec moby2 ping 192.168.0.1 -c 1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.037 ms

--- 192.168.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.037/0.037/0.037/0.000 ms

v

Three-containers w/ OVS bridge

구성



1
2
3
4
5



moby1: eth1 192.168.0.1/24
  | (ovs-br1)
moby2: eth1 - (moby2-bridge) - eth2
  | (ovs-br2)
moby3: eth1 192.168.0.2/24

OVS 브릿지 생성



1
2
3
4
5
6
7
8
9
10
11
12
13



vagrant@localhost ~]$ sudo ovs-vsctl add-br ovs-br1
vagrant@localhost ~]$ sudo ovs-vsctl add-br ovs-br2
vagrant@localhost ~]$ sudo ovs-vsctl show
94185d0-937e-49bf-a0a4-69ca5027718b
   Bridge "ovs-br2"
       Port "ovs-br2"
           Interface "ovs-br2"
               type: internal
   Bridge "ovs-br1"
       Port "ovs-br1"
           Interface "ovs-br1"
               type: internal
   ovs_version: "2.9.0"

컨테이너 생성후 ovs-bridge에 연결



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20



[vagrant@localhost ovs_3hosts]$ cat run.sh
sudo ovs-vsctl add-br ovs-br1
sudo ovs-vsctl add-br ovs-br2
sudo ovs-vsctl show

docker run -dit --name=moby1 ubuntu
docker exec moby1 apt-get update
docker exec moby1 apt-get -y install iproute2 iputils-ping
sudo ovs-docker add-port ovs-br1 eth1 moby1 --ipaddress=192.168.0.1/24

docker run -dit --privileged --name=moby2 ubuntu
docker exec moby2 apt-get update
docker exec moby2 apt-get -y install iproute2 iputils-ping
sudo ovs-docker add-port ovs-br1 eth1 moby2
sudo ovs-docker add-port ovs-br2 eth2 moby2

docker run -dit --name=moby3 ubuntu
docker exec moby3 apt-get update
docker exec moby3 apt-get -y install iproute2 iputils-ping
sudo ovs-docker add-port ovs-br2 eth1 moby3 --ipaddress=192.168.0.2/24

host-bridge

moby2안에서 bridge를 생성하여야 하는데, 기본적으로 NET-ADMIN capability를 가지고 있지 못하므로 아래와 같이 실패할 수 있습니다.



1
2



root@79c82193d876:/# ip link add name moby2-bridge type bridge
RTNETLINK answers: Operation not permitted

그래서 위에서 moby2는 --privileged 옵션을 이용하여 구동하였습니다. 자 브릿지를 만들어 봅시다.



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16



[vagrant@localhost ovs_3hosts]$ docker exec -it moby2 ip link add name moby2-bridge type bridge
[vagrant@localhost ovs_3hosts]$ docker exec -it moby2 ip link set dev moby2-bridge up
[vagrant@localhost ovs_3hosts]$ docker exec -it moby2 ip link set dev eth1 master moby2-bridge
[vagrant@localhost ovs_3hosts]$ docker exec -it moby2 ip link set dev eth2 master moby2-bridge

[vagrant@localhost ovs_3hosts]$ docker exec -it moby2 ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: moby2-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 1e:d3:03:6a:8d:53 brd ff:ff:ff:ff:ff:ff
55: eth0@if56: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
57: eth1@if58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master moby2-bridge state UP mode DEFAULT group default qlen 1000
    link/ether 1e:d3:03:6a:8d:53 brd ff:ff:ff:ff:ff:ff link-netnsid 0
59: eth2@if60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master moby2-bridge state UP mode DEFAULT group default qlen 1000
    link/ether 7e:a1:31:29:09:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0

컨테이너간 연결 확인



1
2
3
4
5
6
7
8
9
10



[vagrant@localhost ~]$ docker exec -it moby2 apt-get install tcpdump

[vagrant@localhost ovs_3hosts]$ docker exec moby1 ping 192.168.0.2 -c 1
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=0.339 ms

--- 192.168.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.339/0.339/0.339/0.000 ms
[vagrant@localhost ovs_3hosts]$

이 때 moby2에서 moby2-bridge에 대고 패킷을 뜨면 위의 연결 확인 패킷을 잡을 수 있습니다.



1
2
3
4
5
6
7



[vagrant@localhost ~]$ docker exec -it moby2 tcpdump -i moby2-bridge -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on moby2-bridge, link-type EN10MB (Ethernet), capture size 262144 bytes
05:24:36.195869 IP 192.168.0.1 > 192.168.0.2: ICMP echo request, id 344, seq 1, length 64
E..T..@.@..............L.X...0.[....G....................... !"#$%&'()*+,-./01234567
05:24:36.195971 IP 192.168.0.2 > 192.168.0.1: ICMP echo reply, id 344, seq 1, length 64
E..T....@.x...........!L.X...0.[....G....................... !"#$%&'()*+,-./01234567

blog ovs docker centos vagrant

Written by
Blog Logo

양철웅

Chul-Woong Yang

Published

Supported by