Kolla OpenStack deployment fails with "haproxy: waiting for virtual IP address"

I am trying to deploy OpenStack Queens with kolla-ansible (7.0.0) on Ubuntu hosts, following the official instructions.

After successful Bootstrap server and precheck the provide Command fails:

RUNNING TRADER [haproxy : Waiting for virtual IP to appear] ************************************************** ** ********
fatal: [testcloudcontrol01]: FAILED! => {"changed": wrong, "expired": 300, "msg": "Waiting timeout on 10.52.41.98:3306"}
fatal: [testcloudcontrol02]: FAILED! => {"changed": wrong, "expired": 300, "msg": "Waiting timeout on 10.52.41.98:3306"}

The reason for the test is that the test fails kolla_internal_vip_address does not come up

globals.yml

config_strategy: "COPY_ALWAYS"
kolla_base_distro: "ubuntu"
kolla_install_type: "binary"
openstack_release: "queens"
kolla_internal_vip_address: "10.52.41.98"
kolla_internal_fqdn: "testcloudapi.example.com"
kolla_external_vip_address: "{{kolla_internal_vip_address}}"
kolla_external_fqdn: "{{kolla_internal_fqdn}}"
Network interface: "ens160"
api_interface: "ens160"
storage_interface: "ens161"
keepalived_virtual_router_id: "148"

I am currently fixated on queens because I want to replicate our production environment for testing.

The issue of IP Address on one of the nodes where Haproxy is to be deployed:

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group preset qlen 1
link / loopback 00: 00: 00: 00: 00: 00 brd 00: 00: 00: 00: 00: 00
inet 127.0.0.1/8 scope lo lo
valid_lft forever Preferred_lft forever
inet6 :: 1/128 Scope Host
valid_lft forever Preferred_lft forever
2: ens160:  mtu 1500 qdisc mq status UP group preset qlen 1000
link / ether 00: 50: 56: a1: 6a: 2c brd ff: ff: ff: ff: ff: ff
inet 10.52.41.100/24 ​​brd 10.52.41.255 Scope global ens160
valid_lft forever Preferred_lft forever
inet6 fe80 :: 250: 56ff: fea1: 6a2c / 64 area connection
valid_lft forever Preferred_lft forever
3: ens161:  mtu 1500 qdisc mq status UP group preset qlen 1000
link / ether 00: 50: 56: a1: 7d: 07 brd ff: ff: ff: ff: ff: ff
inet 10.52.42.100/24 ​​brd 10.52.42.255 Scope global ens161
valid_lft forever Preferred_lft forever
inet6 fe80 :: 250: 56ff: fea1: 7d07 / 64 Area link
valid_lft forever Preferred_lft forever
4: ens224:  mtu 1500 qdisc mq status UP group preset qlen 1000
link / ether 00: 50: 56: a1: 23: 6d brd ff: ff: ff: ff: ff: ff
inet 10.52.40.100/24 ​​brd 10.52.40.255 Scope global ens224
valid_lft forever Preferred_lft forever
inet6 fe80 :: 250: 56ff: fea1: 236e / 64 Area Connection
valid_lft forever Preferred_lft forever
5: ens256:  mtu 1500 qdisc mq status UP group preset qlen 1000
link / ether 00: 50: 56: a1: 20: 12 brd ff: ff: ff: ff: ff: ff
inet 10.52.44.100/24 ​​brd 10.52.44.255 Scope global ens256
valid_lft forever Preferred_lft forever
inet6 fe80 :: 250: 56ff: fea1: 2012/64 link to the section
valid_lft forever Preferred_lft forever
6: docker0:  mtu 1500 qdisc noqueue state DOWN group preset
link / ether 02: 42: b0: 8a: 93: e7 brd ff: ff: ff: ff: ff: ff
Inet 172.17.0.1/16 Scope of Global Docker0
valid_lft forever Preferred_lft forever

The nodes are VMware virtual machines with VMXNet3 network.

Issue of Docker logs are preserved:

+ sudo -E kolla_set_configs
INFO: __ main __: Loading the configuration file under /var/lib/kolla/config_files/config.json
INFO: __ main __: Checks the configuration file
INFO: __ main __: The Kolla configuration strategy is set to COPY_ALWAYS
INFO: __ main __: Copy service configuration files
INFO: __ main __: Delete /etc/keepalived/keepalived.conf
INFO: __ main __: Copy from /var/lib/kolla/config_files/keepalived.conf to /etc/keepalived/keepalived.conf
INFO: __ main __: Set the permission for /etc/keepalived/keepalived.conf
INFO: __ main __: command to execute
++ cat / run_command
+ CMD = & # 39; / usr / sbin / keepalived -nld -p /run/keepalived.pid'
+ ARGS =
+ [[ ! -n '' ]]+. kolla_extend_start
++ modprobe ip_vs
++ & # 39;[' -f /run/keepalived.pid ']& # 39;
+ echo & # 39; command executed: & # 39;  & # 39; & # 39; / usr / sbin / keepalived -nld -p /run/keepalived.pid & # 39;  & # 39; & # 39; & # 39;
Running command: & # 39; / usr / sbin / keepalived -nld -p /run/keepalived.pid'
+ exec / usr / sbin / keepalived -nld -p / run / keepalived.pid
Thu Dec 13 12:10:26 2018: Starting Keepalived Version 1.3.9 (10 / 21,2017)
Thu Dec 13 12:10:26 2018: Opening file & # 39; /etc/keepalived/keepalived.conf&#39 ;.
Thu Dec 13 12:10:26 2018: Start of the Healthcheck children's process, pid = 11
Thu Dec 13 12:10:26 2018: Opening file & # 39; /etc/keepalived/keepalived.conf&#39 ;.
Thu Dec 13 12:10:26 2018: VRRP sub-process is started, pid = 12
Thu Dec 13 12:10:26 2018: ------< Global definitions >------
Thu Dec 13 12:10:26 2018: Router ID = testcloudcontrol01.example.com
Thu Dec 13 12:10:26 2018: standard interface = eth0
Thu Dec 13 12:10:26 2018: LVS flush = wrong
Thu Dec 13 12:10:26 2018: VRRP IPv4 Mcast Group = 224.0.0.18
Thu Dec 13 12:10:26 2018: VRRP IPv6 Mcast Group = ff02 :: 12
Thu Dec 13 12:10:26 2018: Indispensable ARP delay = 5
Thu Dec 13 12:10:26 2018: Free ARP repetition = 5
Thu Dec 13 12:10:26 2018: Indispensable ARP update timer = 0
Thu Dec 13 12:10:26 2018: Repeat ARP update retry = 1
Thu Dec 13 12:10:26 2018: Indispensable lower priority ARP delay = 4294
Thu Dec 13 12:10:26 2018: Indispensable ARP repetition with lower priority = -1
Thu Dec 13 12:10:26 2018: Submit the ad after receiving a lower priority
Thu 13.12. 12:10:26 2018: Submit an ad after the higher priority = false
Thu Dec 13 12:10:26 2018: Free ARP interval = 0
Thu Dec 13 12:10:26 2018: Unnecessary NA interval = 0
Thu Dec 13 12:10:26 2018: VRRP standard protocol version = 2
Thu Dec 13 12:10:26 2018: Iptables input string = INPUT
Thu Dec 13 12:10:26 2018: With ipsets = true
Thu Dec 13 12:10:26 2018: ipset IPv4 address set = Keepalived
Thu Dec 13 12:10:26 2018: ipset IPv6 address set = keepalived6
Thu Dec 13 12:10:26 2018: ipset IPv6 address, iface set = keepalived_if6
Thu Dec 13 12:10:26 2018: VRRP check unicast_src = false
Thu Dec 13 12:10:26 2018: VRRP Skip the adresses = wrong
Thu Dec 13 12:10:26 2018: VRRP strict mode = wrong
Thu Dec 13 12:10:26 2018: VRRP process priority = 0
Thu Dec 13 12:10:26 2018: VRRP do not swap = wrong
Thu Dec 13 12:10:26 2018: Checker process priority = 0
Thu Dec 13 12:10:26 2018: Checker do not exchange = wrong
Thu Dec 13 12:10:26 2018: SNMP has been deactivated
Thu Dec 13 12:10:26 2018: SNMP Checker disabled
Thu Dec 13 12:10:26 2018: SNMP RFCv2 disabled
Thu Dec 13 12:10:26 2018: SNMP RFCv3 disabled
Thu Dec 13 12:10:26 2018: SNMP traps disabled
Thu Dec 13 12:10:26 2018: SNMP-Socket = Default (Unix: / var / agentx / master)
Thu Dec 13 12:10:26 2018: Network namespace = (default)
Thu Dec 13 12:10:26 2018: DBus disabled
Thu Dec 13 12:10:26 2018: DBus service name = (null)
Thu Dec 13 12:10:26 2018: Script security disabled
Thu Dec 13 12:10:26 2018: Default script uid: gid 0: 0
Thu Dec 13 12:10:26 2018: Register the kernel-netlink-reflector
Thu Dec 13 12:10:26 2018: Register the kernel netlink command channel
Thu Dec 13 12:10:26 2018: Register a free ARP channel
Thu Dec 13 12:10:26 2018: Opening file & # 39; /etc/keepalived/keepalived.conf&#39 ;.
Thu Dec 13 12:10:26 2018: WARNING - The default user & # 39; keepalived_script & # 39; for the script execution does not exist. Please create.
Thu Dec 13 12:10:26 2018: Auth_pass is truncated to 8 characters
Thu Dec 13 12:10:26 2018: SECURITY INFRINGEMENT - Scripts are running, but script_security is not enabled.
Thu Dec 13 12:10:26 2018: ------< Global definitions >------
Thu Dec 13 12:10:26 2018: Router ID = testcloudcontrol01.example.com
Thu Dec 13 12:10:26 2018: standard interface = eth0
Thu Dec 13 12:10:26 2018: LVS flush = wrong
Thu Dec 13 12:10:26 2018: VRRP IPv4 Mcast Group = 224.0.0.18
Thu Dec 13 12:10:26 2018: VRRP IPv6 Mcast Group = ff02 :: 12
Thu Dec 13 12:10:26 2018: Indispensable ARP delay = 5
Thu Dec 13 12:10:26 2018: Free ARP repetition = 5
Thu Dec 13 12:10:26 2018: Indispensable ARP update timer = 0
Thu Dec 13 12:10:26 2018: Repeat ARP update retry = 1
Thu Dec 13 12:10:26 2018: Indispensable lower priority ARP delay = 5
Thu Dec 13 12:10:26 2018: Repeat lower priority ARP = 5
Thu Dec 13 12:10:26 2018: Submit the ad after receiving a lower priority
Thu 13.12. 12:10:26 2018: Submit an ad after the higher priority = false
Thu Dec 13 12:10:26 2018: Free ARP interval = 0
Thu Dec 13 12:10:26 2018: Unnecessary NA interval = 0
Thu Dec 13 12:10:26 2018: VRRP standard protocol version = 2
Thu Dec 13 12:10:26 2018: Iptables input string = INPUT
Thu Dec 13 12:10:26 2018: With ipsets = false
Thu Dec 13 12:10:26 2018: ipset IPv4 address set = Keepalived
Thu Dec 13 12:10:26 2018: ipset IPv6 address set = keepalived6
Thu Dec 13 12:10:26 2018: ipset IPv6 address, iface set = keepalived_if6
Thu Dec 13 12:10:26 2018: VRRP check unicast_src = false
Thu Dec 13 12:10:26 2018: VRRP Skip the adresses = wrong
Thu Dec 13 12:10:26 2018: VRRP strict mode = wrong
Thu Dec 13 12:10:26 2018: VRRP process priority = 0
Thu Dec 13 12:10:26 2018: VRRP do not swap = wrong
Thu Dec 13 12:10:26 2018: Checker process priority = 0
Thu Dec 13 12:10:26 2018: Checker do not exchange = wrong
Thu Dec 13 12:10:26 2018: SNMP has been deactivated
Thu Dec 13 12:10:26 2018: SNMP Checker disabled
Thu Dec 13 12:10:26 2018: SNMP RFCv2 disabled
Thu Dec 13 12:10:26 2018: SNMP RFCv3 disabled
Thu Dec 13 12:10:26 2018: SNMP traps disabled
Thu Dec 13 12:10:26 2018: SNMP-Socket = Default (Unix: / var / agentx / master)
Thu Dec 13 12:10:26 2018: Network namespace = (default)
Thu Dec 13 12:10:26 2018: DBus disabled
Thu Dec 13 12:10:26 2018: DBus service name = (null)
Thu Dec 13 12:10:26 2018: Script security disabled
Thu Dec 13 12:10:26 2018: Default script uid: gid 0: 0
Thu Dec 13 12:10:26 2018: ------< VRRP Topology >------
Thu Dec 13 12:10:26 2018: VRRP instance = kolla_internal_vip_148
Thu Dec 13 12:10:26 2018: Using VRRPv2
Thu Dec 13 12:10:26 2018: Want State = BACKUP
Thu Dec 13 12:10:26 2018: Runs on device = ens160
Thu Dec 13 12:10:26 2018: Skip the display of IP addresses = no
Thu Dec 13 12:10:26 2018: Enforce strict VRRP compliance = no
Thu Dec 13 12:10:26 2018: Using src_ip = 10.52.41.100
Thu Dec 13 12:10:26 2018: Indispensable ARP delay = 5
Thu Dec 13 12:10:26 2018: Free ARP repetition = 5
Thu Dec 13 12:10:26 2018: Indispensable ARP update timer = 0
Thu Dec 13 12:10:26 2018: Repeat ARP update retry = 1
Thu Dec 13 12:10:26 2018: Indispensable lower priority ARP delay = 5
Thu Dec 13 12:10:26 2018: Repeat lower priority ARP = 5
Thu Dec 13 12:10:26 2018: Submit the ad after receiving a lower priority
Thu 13.12. 12:10:26 2018: Submit an ad after the higher priority = false
Thu Dec 13 12:10:26 2018: Virtual Router ID = 148
Thu Dec 13 12:10:26 2018: Priority = 1
Thu Dec 13 12:10:26 2018: Display interval = 1 sec
Thu Dec 13 12:10:26 2018: Accept activated
Thu Dec 13 12:10:26 2018: Disability disabled
Thu Dec 13 12:10:26 2018: Promotion_Secondary disabled
Thu Dec 13 12:10:26 2018: Authentication type = SIMPLE_PASSWORD
Thu Dec 13 12:10:26 2018: Password = 0RXbQYFF
Thu Dec 13 12:10:26 2018: Tracked scripts = 1
Thu Dec 13 12:10:26 2018: check_alive weight 0
Thu Dec 13 12:10:26 2018: Virtual IP = 1
Thu Dec 13 12:10:26 2018: 10.52.41.98/32 dev ens160 Scope global
Thu Dec 13 12:10:26 2018: ------< VRRP Scripts >------
Thu Dec 13 12:10:26 2018: VRRP script = check_alive
Thu Dec 13 12:10:26 2018: Command = /check_alive.sh
Thu Dec 13 12:10:26 2018: Interval = 2 sec
Thu Dec 13 12:10:26 2018: Timeout = 0 sec
Thu Dec 13 12:10:26 2018: Weight = 0
Thu Dec 13 12:10:26 2018: Ascent = 10
Thu Dec 13 12:10:26 2018: Case = 2
Thu Dec 13 12:10:26 2018: Uncertain = No
Thu Dec 13 12:10:26 2018: Status = INIT
Thu Dec 13 12:10:26 2018: Script uid: gid = 0: 0
Thu Dec 13 12:10:26 2018: ------< NIC >------
Thu Dec 13 12:10:26 2018: Name = see
Thu Dec 13 12:10:26 2018: Index = 1
Thu Dec 13 12:10:26 2018: IPv4 address = 127.0.0.1
Thu Dec 13 12:10:26 2018: IPv6 address = ::
Thu Dec 13 12:10:26 2018: is UP
Thu Dec 13 12:10:26 2018: RUNNING
Thu Dec 13 12:10:26 2018: MTU = 65536
Thu Dec 13 12:10:26 2018: HW-Type = LOOPBACK
Thu Dec 13 12:10:26 2018: ------< NIC >------
Thu Dec 13 12:10:26 2018: Name = ens160
Thu Dec 13 12:10:26 2018: Index = 2
Thu Dec 13 12:10:26 2018: IPv4 address = 10.52.41.100
Thu Dec 13 12:10:26 2018: IPv6 address = fe80 :: 250: 56ff: fea1: 6a2c
Thu Dec 13 12:10:26 2018: MAC = 00: 50: 56: a1: 6a: 2c
Thu Dec 13 12:10:26 2018: is UP
Thu Dec 13 12:10:26 2018: RUNNING
Thu Dec 13 12:10:26 2018: MTU = 1500
Thu Dec 13 12:10:26 2018: HW type = ETHERNET
Thu Dec 13 12:10:26 2018: ------< NIC >------
Thu Dec 13 12:10:26 2018: Name = ens161
Thu Dec 13 12:10:26 2018: Index = 3
Thu Dec 13 12:10:26 2018: IPv4 address = 10.52.42.100
Thu Dec 13 12:10:26 2018: IPv6 address = fe80 :: 250: 56ff: fea1: 7d07
Thu Dec 13 12:10:26 2018: MAC = 00: 50: 56: a1: 7d: 07
Thu Dec 13 12:10:26 2018: is UP
Thu Dec 13 12:10:26 2018: RUNNING
Thu Dec 13 12:10:26 2018: MTU = 1500
Thu Dec 13 12:10:26 2018: HW type = ETHERNET
Thu Dec 13 12:10:26 2018: ------< NIC >------
Thu Dec 13 12:10:26 2018: Name = ens224
Thu Dec 13 12:10:26 2018: Index = 4
Thu Dec 13 12:10:26 2018: IPv4 address = 10.52.40.100
Thu Dec 13 12:10:26 2018: IPv6 address = fe80 :: 250: 56ff: fea1: 236e
Thu Dec 13 12:10:26 2018: MAC = 00: 50: 56: a1: 23: 6e
Thu Dec 13 12:10:26 2018: is UP
Thu Dec 13 12:10:26 2018: RUNNING
Thu Dec 13 12:10:26 2018: MTU = 1500
Thu Dec 13 12:10:26 2018: HW type = ETHERNET
Thu Dec 13 12:10:26 2018: ------< NIC >------
Thu Dec 13 12:10:26 2018: Name = ens256
Thu Dec 13 12:10:26 2018: Index = 5
Thu Dec 13 12:10:26 2018: IPv4 address = 10.52.44.100
Thu Dec 13 12:10:26 2018: IPv6 address = fe80 :: 250: 56ff: fea1: 2012
Thu Dec 13 12:10:26 2018: MAC = 00: 50: 56: a1: 20: 12
Thu Dec 13 12:10:26 2018: is UP
Thu Dec 13 12:10:26 2018: RUNNING
Thu Dec 13 12:10:26 2018: MTU = 1500
Thu Dec 13 12:10:26 2018: HW type = ETHERNET
Thu Dec 13 12:10:26 2018: ------< NIC >------
Thu Dec 13 12:10:26 2018: Name = Docker0
Thu Dec 13 12:10:26 2018: Index = 6
Thu Dec 13 12:10:26 2018: IPv4 address = 172.17.0.1
Thu Dec 13 12:10:26 2018: IPv6 address = ::
Thu Dec 13 12:10:26 2018: MAC = 02: 42: B0: 8a: 93: e7
Thu Dec 13 12:10:26 2018: is UP
Thu Dec 13 12:10:26 2018: MTU = 1500
Thu Dec 13 12:10:26 2018: HW type = ETHERNET
Thu Dec 13 12:10:26 2018: Using the LinkWatch Kernel Netlink Reflector ...
Thu Dec 13 12:10:26 2018: VRRP_Instance (kolla_internal_vip_148) Enter BACKUP STATUS
Thu Dec 13 12:10:26 2018: /check_alive.sh ended with status 1
Thu Dec 13 12:10:28 2018: /check_alive.sh ended with status 1
Thu Dec 13 12:10:30 2018: VRRP_Instance (kolla_internal_vip_148) Now in error state
Thu Dec 13 12:10:30 2018: /check_alive.sh ended with status 1
Thu Dec 13 12:10:32 2018: /check_alive.sh ended with status 1
[message repeats until I stop the container]

That's it, the two keepalived instances remain in FAULT status, the IP address is not enabled on any of the VMs.

I have read this question and the answer, although I have no error messages in the log files:

  • keepalived_virtual_router_id has been changed and is unique
  • I ran kolla-genpwd once again. I confirmed that keepalived_password is employed /etc/kolla/passwords.yml
  • kolla_internal_vip_address is reachable from Network Interface, The main IP of this interface is on the same network. I can manually set the additional IP address and it works.
  • kolla-ansible prechecks passes
  • Selinux is not active on Ubuntu

On the hypervisor page, I tried activating Promiscuous mode for the port group of this interface. That did not make a difference.