OpenStack

Neutron lbaas代理https实践

Posted by 爱折腾的工程师 on Sunday, November 19, 2017

背景

通过neutron-lbaas实现对https的代理，引用官方的解释 https://docs.openstack.org/mitaka/networking-guide/config-lbaas.html，neutron-lbaas是OpenStack负载均衡服务的实现，有lbaas v1和lbaas v2两种实现，其中v1是在Juno版本引入，在Liberty版本被弃用；v2是在Kilo版本引入；v1和v2无法兼容，两种实现都使用agent。agent处理HAProxy配置并管理HAProxy守护进程。另一种基于LBaaS v2的实现是Octavia项目，具有独立的API和独立的工作进程，在nova创建的虚拟机中创建负载均衡器。对于Octavia，不再需要agent。

测试环境

CentOS 7.1 （OpenStack Kilo）

Neutron lbaas v2（Kilo）

Barbican（Kilo）

概念介绍

lbaas有一些名字概念，引用官方的解释，它们分别是：

  • Load balancer 负载均衡器会占用一个子网的端口和ip.

  • Listener 负载平衡器可以监听多个端口上的请求。每一个端口都由Listener来指定.

  • pool 池包含通过负载均衡器提供服务的成员列表.

  • member 成员是在负载均衡器后端真正提供服务的server。 每个成员由提供服务的IP地址和端口来指定.

  • Health monitor 成员可能会不时地掉线，健康监视器会将服务没有正常响应的成员转移出去。健康监视器与池相关联。

LBaaS v2通过不同的服务插件有多个实现。 两个最常见的实现使用代理或Octavia服务。 这两个实现都使用LBaaS v2 API。

LBaaS v2将listener的概念添加到负载均衡器，LBaaS v2允许在单个负载平衡器IP地址上配置多个listener。

用官方的一张图来看它们之间的关系

部署lbaas

这里只介绍lbaas v2后端为haproxy，代理https的方式.

安装lbaas

[root@con01 ~(keystone_admin)]# yum install -y openstack-neutron-lbaas haproxy

配置lbaas v2

[root@con01 ~(keystone_admin)]# egrep -v "^$|^#" /etc/neutron/lbaas_agent.ini
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
device_driver = neutron_lbaas.drivers.haproxy.namespace_driver.HaproxyNSDriver
#device_driver = neutron_lbaas.services.loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver # 默认值是这个，误导性太大
[haproxy]
user_group = haproxy

[root@con01 ~(keystone_admin)]# egrep -v "^$|^#" /etc/neutron/neutron_lbaas.conf
[DEFAULT]
[quotas]
[service_providers]
service_provider=LOADBALANCERV2:Haproxy:neutron_lbaas.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
[certificates]

# 追加lbaas v2 plugin
[root@con01 ~(keystone_admin)]# egrep -v "^$|^#" /etc/neutron/neutron.conf
[DEFAULT]
service_plugins = router,neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2
[keystone_authtoken]
auth_uri = http://10.125.224.21:35357/v2.0
identity_uri = http://10.125.224.52:35357/
admin_tenant_name = services
admin_user = neutron
admin_password = xxxxxx
auth_version = v2

初始化lbaas数据库表

[root@con01 ~(keystone_admin)]# neutron-db-manage --service lbaas upgrade head

创建lbaas v2启动服务脚本

[root@con01 ~(keystone_admin)]# egrep -v "^$|^#" /usr/lib/systemd/system/neutron-lbaasv2-agent.service
[Unit]
Description=OpenStack Neutron Load Balancing V2 as a Service Agent
After=syslog.target network.target
[Service]
Type=simple
User=root
ExecStart=/usr/bin/neutron-lbaasv2-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/neutron_lbaas.conf --config-file /etc/neutron/lbaas_agent.ini --log-file /var/log/neutron/lbaasv2-agent.log
PrivateTmp=false
KillMode=process
[Install]
WantedBy=multi-user.target

启动服务

[root@con01 ~(keystone_admin)]# systemctl daemon-reload
[root@con01 ~(keystone_admin)]# systemctl start neutron-lbaasv2-agent.service

[root@con01 ~(keystone_admin)]# systemctl restart neutron-server.service

lbaas https方式

lbaas v2支持TERMINATED_HTTPS和HTTPS，后端driver为haproxy的话，分别对应haproxy代理ssl的这两种模式：

1、haproxy本身提供ssl证书，以http访问后端realserver（lbaas TERMINATED_HTTPS）

这种方式，haproxy需要支持ssl(从haproxy 1.5版本开始支持ssl)，对应的haproy配置如下：

  frontend https_frontend
  bind *:443 ssl crt /etc/ssl/certs/xxxx.pem
  mode http
  option httpclose
  option forwardfor
  reqadd X-Forwarded-Proto:\ https
  default_backend real_server

  backend real_server
  mode http
  balance roundrobin
  server s1 192.168.1.5:80
  server s2 192.168.1.6:80

上述方式还需要pem文件，pem是private key和certificate的合体；好在lbaas key管理后端backend支持barbican和local，这里只介绍barbican方式。

2、haproxy本身只提供代理，以https访问后端realserver，简称ssl透传（lbaas HTTPS）

这种方式，haproxy支不支持ssl都没关系; 因为是tcp方式，所以后端realserver就获取不到报头X-Forwarded-*信息。对应的haproxy配置如下：

  frontend https_frontend
  bind *:443
  mode tcp
  default_backend real_server

  backend real_server
  mode tcp
  balance roundrobin
  server node01 192.168.1.5:443
  server node02 192.168.1.6:443

部署barbican

barbican安装

参考这篇文章 https://github.com/cloudkeep/barbican/wiki/Barbican-Quick-Start-Guide, 推荐使用virtualenv虚拟环境来安装barbican

yum install python-devel libffi-devel openssl-devel openldap-devel gcc -y  # 安装相关依赖包

pip install virtualenv
virtualenv barbican27
source barbican27/bin/activate   # 进入虚拟环境；推出虚拟环境是deactivate

pip install MySQL-python
git clone https://github.com/openstack/barbican.git ／opt
git checkout kilo-eol
cd barbican
bin/barbican.sh install

与keystone集成、配置详情见devstack kilo barbican过程 https://github.com/openstack/barbican/blob/kilo-eol/contrib/devstack/lib/barbican

barbican集成keystone

keystone user-create --name=barbican --pass=barbican --tenant-id <services租户ID> --email=barbican@example.com
keystone user-role-add  --tenant-id <services租户ID> --user-id <barbican用户ID> --role-id <admin角色ID>
keystone service-create --name=barbican --type='key-manager' --description="Barbican Service"
keystone endpoint-create --region NEW_Region \
    --service-id <barbican服务ID> \
    --publicurl "http://10.125.224.21:9311" \
    --adminurl "http://10.125.224.21:9312" \
    --internalurl "http://10.125.224.21:9311"

创建barbican数据库

create database barbican character set utf8;  # 进入mariadb，创建barbican数据库
grant all privileges on barbican.* to barbican@'localhost' identified by 'barbican';
grant all privileges on barbican.* to barbican@'%' identified by 'barbican';

修改barbican配置

export BARBICAN_CONF_DIR=/etc/barbican
export BARBICAN_DIR=/opt/barbican

mkdir -p $BARBICAN_CONF_DIR
cp $BARBICAN_DIR/etc/barbican/barbican-api.conf $BARBICAN_CONF_DIR
cp $BARBICAN_DIR/etc/barbican/barbican-api-paste.ini $BARBICAN_CONF_DIR
cp $BARBICAN_DIR/etc/barbican/barbican-admin-paste.ini $BARBICAN_CONF_DIR
cp -R $BARBICAN_DIR/etc/barbican/vassals $BARBICAN_CONF_DIR/
cp $BARBICAN_DIR/etc/barbican/barbican-functional.conf $BARBICAN_CONF_DIR
cp $BARBICAN_DIR/etc/barbican/policy.json $BARBICAN_CONF_DIR

touch /var/log/barbican/api.log
mkdir -p /var/lib/barbican/cache

[root@con01 ~(keystone_admin)]# egrep -v "^$|^#" /etc/barbican/barbican-api.conf
[DEFAULT]
verbose = True
debug = False
bind_host = 0.0.0.0
bind_port = 9311
host_href = http://10.125.224.21:9311   # 根据节点ip修改
log_file = /var/log/barbican/api.log    
backlog = 4096
max_allowed_secret_in_bytes = 10000
max_allowed_request_size_in_bytes = 1000000
sql_connection = mysql://barbican:barbican@con01/barbican?charset=utf8
sql_idle_timeout = 3600
default_limit_paging = 10
max_limit_paging = 100
workers = 1
delayed_delete = False
scrub_time = 43200
scrubber_datadir = /var/lib/barbican/scrubber
policy_file=/etc/barbican/policy.json
policy_default_rule=default
ampq_durable_queues = True
rabbit_userid=guest
rabbit_password=guest
rabbit_ha_queues = True
rabbit_port=5672
rabbit_hosts=con01:5672   # 根据节点增加
[queue]
enable = False
namespace = 'barbican'
topic = 'barbican.workers'
version = '1.1'
server_name = 'barbican.queue'
[retry_scheduler]
initial_delay_seconds = 10.0
periodic_interval_max_seconds = 10.0
[keystone_notifications]
enable = False
control_exchange = 'openstack'
topic = 'notifications'
allow_requeue = False
version = '1.0'
thread_pool_size = 10
[secrets]
broker = rabbit://guest:guest@con01    # 视情况修改
[secretstore]
namespace = barbican.secretstore.plugin
enabled_secretstore_plugins = store_crypto
[crypto]
namespace = barbican.crypto.plugin
enabled_crypto_plugins = simple_crypto
[simple_crypto_plugin]
kek = 'YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY='
[dogtag_plugin]
pem_path = '/etc/barbican/kra_admin_cert.pem'
dogtag_host = localhost
dogtag_port = 8443
nss_db_path = '/etc/barbican/alias'
nss_db_path_ca = '/etc/barbican/alias-ca'
nss_password = 'password123'
simple_cmc_profile = 'caOtherCert'
[p11_crypto_plugin]
library_path = '/usr/lib/libCryptoki2_64.so'
login = 'mypassword'
mkek_label = 'an_mkek'
mkek_length = 32
hmac_label = 'my_hmac_label'
[kmip_plugin]
username = 'admin'
password = 'password'
host = localhost
port = 5696
keyfile = '/path/to/certs/cert.key'
certfile = '/path/to/certs/cert.crt'
ca_certs = '/path/to/certs/LocalCA.crt'
[certificate]
namespace = barbican.certificate.plugin
enabled_certificate_plugins = simple_certificate
[certificate_event]
namespace = barbican.certificate.event.plugin
enabled_certificate_event_plugins = simple_certificate

[root@con01 ~(keystone_admin)]# vim $BARBICAN_CONF_DIR/vassals/barbican-api.ini
[uwsgi]
buffer-size = 65535  # 增加这个配置

[root@con01 ~(keystone_admin)]# vim /etc/barbican/barbican-api-paste.ini # 修改为如下配置
[pipeline:barbican_api]
pipeline = keystone_authtoken context apiapp
[filter:keystone_authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
auth_protocol = http
auth_host = 10.125.224.21
auth_port = 35357
auth_uri = http://10.125.224.21:5000
identity_uri = http://10.125.224.21:35357
admin_tenant_name = services
admin_user = barbican
admin_password = barbican
auth_version = v2
signing_dir = /var/lib/barbican/cache

[root@con01 ~(keystone_admin)]# vim /etc/barbican/policy.json # 修改policy权限,增加rule:admin和普通角色_member_,对应到keystone有的role
"secret:decrypt": "rule:secret_decrypt_non_private_read or rule:secret_creator_user or rule:secret_acl_read or rule:admin or rule:_member_",
"secret:get": "rule:secret_non_private_read or rule:secret_creator_user or rule:secret_acl_read or rule:admin or rule:_member_",
"secrets:get": "rule:all_but_audit or rule:admin or rule:_member_",
"container:get": "rule:container_non_private_read or rule:container_creator_user or rule:container_acl_read or rule:admin or rule:_member_",

启动barbican api

[root@con01 ~(keystone_admin)]# source barbican27/bin/activate
(barbican27) [root@con01 ~(keystone_admin)]# uwsgi --master --emperor /etc/barbican/vassals/

更新barbicanclient

git clone https://github.com/openstack/python-barbicanclient /opt/
git checkout kilo-eol
mv /usr/lib/python2.7/site-packages/{barbicanclient,barbicanclient.bak}
cp -r /opt/python-barbicanclient/barbicanclient /usr/lib/python2.7/site-packages/

测试

创建https的负载均衡器来验证，参考官方wiki的这篇教程，很详细。 https://wiki.openstack.org/wiki/Network/LBaaS/docs/how-to-create-tls-loadbalancer#Update_neutron_config

创建认证串和key

openssl genrsa -des3 -out ca.key 1024
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt  
openssl x509  -in  ca.crt -out ca.pem
openssl genrsa -des3 -out ca-int_encrypted.key 1024
openssl rsa -in ca-int_encrypted.key -out ca-int.key
openssl req -new -key ca-int.key -out ca-int.csr -subj "/CN=ca-int@acme.com"
openssl x509 -req -days 3650 -in ca-int.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out ca-int.crt
openssl genrsa -des3 -out server_encrypted.key 1024
openssl rsa -in server_encrypted.key -out server.key
openssl req -new -key server.key -out server.csr -subj "/CN=server@acme.com"
openssl x509 -req -days 3650 -in server.csr  -CA ca-int.crt -CAkey ca-int.key -set_serial 01 -out server.crt

创建Barbican secrets和containers

barbican secret store --payload-content-type='text/plain' \
                      --name='certificate' \
                      --payload="$(cat server.crt)"

barbican secret store --payload-content-type='text/plain' \
                      --name='private_key' \
                      --payload="$(cat server.key)"

barbican container create --name='tls_container' \
                          --type='certificate' --secret="certificate=$(barbican secret list | awk '/ certificate / {print $2}')" \
                          --secret="private_key=$(barbican secret list | awk '/ private_key / {print $2}')"

创建实例和网络

  • 创建两个实例，ip分别为10.10.10.20、10.10.10.21
  • 创建一个私网的子网subet-net01

创建loadbalancer

neutron lbaas-loadbalancer-create --name lb1 subet-net01

创建listener

neutron lbaas-listener-create --loadbalancer lb1 \
          --protocol-port 443 \
          --protocol TERMINATED_HTTPS \
          --name listener1 \
          --default-tls-container-id=$(barbican container list | awk '/ tls_container / {print $2}')

创建pool

neutron lbaas-pool-create --name pool1 --protocol HTTP --listener listener1 --lb-algorithm ROUND_ROBIN

创建members

neutron lbaas-member-create --subnet subet-net01 \
      --address 10.10.10.20 \
      --protocol-port 80 pool1

neutron lbaas-member-create --subnet subet-net01 \
      --address 10.10.10.21 \
      --protocol-port 80 pool1

创建healthmonitor

neutron lbaas-healthmonitor-create --type HTTP  \
        --delay 3 \
        --max-retries 3 \
        --timeout 3 \
        --pool <pool-id>

.pem文件默认会在形如这样的目录下：/var/lib/neutron/lbaas/v2/b2fa71b0-67ba-4e72-b206-fbc306c6d5dc/b6c834e0-5d37-457e-a669-17b683efb176/

设置lbaas v2配额

neutron quota-update --tenant-id TENANT_UUID --loadbalancer 25
neutron quota-update --tenant-id TENANT_UUID --pool 50

代码调试

部分lbaas相关关键代码：

neutron api入口：
neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2

lbaas v2 agent入口:
neutron_lbaas.drivers.haproxy.namespace_driver.HaproxyNSDriver


/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/haproxy/namespace_driver.py(203)create()
199  	    def create(self, loadbalancer):
200  	        namespace = get_ns_name(loadbalancer.id)
201
202  	        self._plug(namespace, loadbalancer.vip_port)    # 创建lbaas namespace
203  ->	      self._spawn(loadbalancer)           		# 根据jinja模版生成haporxy配置文件，并在对应namespace中启动haproxy进程

参考链接

「真诚赞赏，手留余香」

爱折腾的工程师

真诚赞赏，手留余香

使用微信扫描二维码完成支付

CATALOG
    FEATURED TAGS
    devops go k8s llm openstack tkestack 练车