中止节点上全部的osd服务node
systemctl stop ceph-osd@20 systemctl stop ceph-osd@21 systemctl stop ceph-osd@22 systemctl stop ceph-osd@23
在mon节点上移除osd数据库
ceph osd out 23 ceph osd out 22 ceph osd out 21 ceph osd out 20 ceph osd crush remove osd.23 ceph osd crush remove osd.22 ceph osd crush remove osd.21 ceph osd crush remove osd.20 ceph auth del osd.23 ceph auth del osd.22 ceph auth del osd.21 ceph auth del osd.20 ceph osd rm 23 ceph osd rm 22 ceph osd rm 21 ceph osd rm 20 ceph osd crush remove bdc8
等待集群自动恢复便可 若是配置文件中有对应osd在某台节点上的配置,须要删掉网络
systemctl stop neutron-openvswitch-agent.service systemctl stop openstack-nova-compute.service systemctl disable neutron-openvswitch-agent.service systemctl disable openstack-nova-compute.service
查看计算节点服务3d
. admin-openrc.sh nova service-list +----+------------------+------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +----+------------------+------+----------+---------+-------+----------------------------+-----------------+ | 1 | nova-cert | bdc2 | internal | enabled | up | 2018-03-22T02:54:42.000000 | - | | 2 | nova-conductor | bdc2 | internal | enabled | up | 2018-03-22T02:54:43.000000 | - | | 3 | nova-consoleauth | bdc2 | internal | enabled | up | 2018-03-22T02:54:42.000000 | - | | 4 | nova-scheduler | bdc2 | internal | enabled | up | 2018-03-22T02:54:40.000000 | - | | 5 | nova-compute | bdc3 | nova | enabled | up | 2018-03-22T02:54:41.000000 | - | | 6 | nova-compute | bdc2 | nova | enabled | up | 2018-03-22T02:54:45.000000 | - | | 7 | nova-compute | bdc4 | nova | enabled | up | 2018-03-22T02:54:42.000000 | - | | 8 | nova-compute | bdc5 | nova | enabled | up | 2018-03-22T02:54:44.000000 | - | | 9 | nova-compute | bdc6 | nova | enabled | up | 2018-03-22T02:54:44.000000 | - | | 10 | nova-compute | bdc7 | nova | enabled | up | 2018-03-22T02:54:46.000000 | - | | 11 | nova-compute | bdc8 | nova | enabled | up | 2018-03-22T02:54:39.000000 | - | +----+------------------+------+----------+---------+-------+----------------------------+-----------------+
删除bdc8上的计算服务code
nova service-disable 11 nova service-delete 11
元数据库中删除ip
use nova; select * from compute_nodes where host='bdc2'\G delete from compute_nodes where host='bdc2'; select * from services where host='bdc2'\G delete from services where host='bdc2';
查看网络服务rem
# neutron agent-list +--------------------------------------+--------------------+------+-------+----------------+---------------------------+ | id | agent_type | host | alive | admin_state_up | binary | +--------------------------------------+--------------------+------+-------+----------------+---------------------------+ | 08bc9d33-3e67-4a06-9223-74be1c969b8a | Open vSwitch agent | bdc2 | :-) | True | neutron-openvswitch-agent | | 19a31ea2-280b-42b4-9f08-c1f99182e826 | Open vSwitch agent | bdc5 | :-) | True | neutron-openvswitch-agent | | 286336ae-22fd-4391-8470-aae559ee5ccb | DHCP agent | bdc3 | :-) | True | neutron-dhcp-agent | | 42ab605e-3d66-402f-b86a-4ef246df2ecf | Open vSwitch agent | bdc8 | :-) | True | neutron-openvswitch-agent | | 4791844d-07f7-43be-8b27-bcfe8d8a7791 | Open vSwitch agent | bdc6 | :-) | True | neutron-openvswitch-agent | | 63c99510-205a-4f94-ac18-61082c6edb44 | L3 agent | bdc3 | :-) | True | neutron-l3-agent | | 830a75fb-c516-4f6f-ae74-e09bfd558291 | Open vSwitch agent | bdc7 | :-) | True | neutron-openvswitch-agent | | 91df9210-0a27-4933-9f03-3ccce6192011 | Open vSwitch agent | bdc4 | :-) | True | neutron-openvswitch-agent | | a9f7bc30-beda-437d-92a8-be64296c57d0 | Open vSwitch agent | bdc3 | :-) | True | neutron-openvswitch-agent | | cf6443f7-ef8e-4738-bfc5-e491e4f89aeb | Metadata agent | bdc3 | :-) | True | neutron-metadata-agent | +--------------------------------------+--------------------+------+-------+----------------+---------------------------+
删除bdc8上的网络服务it
# neutron agent-delete 42ab605e-3d66-402f-b86a-4ef246df2ecf Deleted agent: 42ab605e-3d66-402f-b86a-4ef246df2ecf
nova库io
MariaDB [nova]> select hypervisor_hostname,deleted_at,deleted from compute_nodes; +---------------------+---------------------+---------+ | hypervisor_hostname | deleted_at | deleted | +---------------------+---------------------+---------+ | bdc3 | NULL | 0 | | bdc2 | NULL | 0 | | bdc4 | NULL | 0 | | bdc5 | NULL | 0 | | bdc6 | NULL | 0 | | bdc7 | NULL | 0 | | bdc8 | 2018-03-22 02:58:53 | 7 | +---------------------+---------------------+---------+ MariaDB [nova]> select * from services where host = 'bdc8'; +---------------------+---------------------+---------------------+----+------+--------------+---------+--------------+----------+---------+-----------------+ | created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | +---------------------+---------------------+---------------------+----+------+--------------+---------+--------------+----------+---------+-----------------+ | 2016-10-13 09:20:41 | 2018-03-22 02:58:49 | 2018-03-22 02:58:53 | 11 | bdc8 | nova-compute | compute | 4442289 | 0 | 11 | NULL | +---------------------+---------------------+---------------------+----+------+--------------+---------+--------------+----------+---------+-----------------+
先在元数据库中删除bdc8的gre信息console
MariaDB [neutron]> select * from ml2_gre_endpoints; +-------------+------+ | ip_address | host | +-------------+------+ | 172.16.13.2 | bdc2 | | 172.16.13.3 | bdc3 | | 172.16.13.4 | bdc4 | | 172.16.13.5 | bdc5 | | 172.16.13.6 | bdc6 | | 172.16.13.7 | bdc7 | | 172.16.13.8 | bdc8 | +-------------+------+ MariaDB [neutron]> delete from ml2_gre_endpoints where ip_address='172.16.13.8';
各个节点中删除链接bdc8的gre隧道
# ovs-vsctl show 8d6ea1e1-9b90-4e6b-94ae-d5a04601daf9 Bridge br-tun fail_mode: secure Port "gre-ac100d03" Interface "gre-ac100d03" type: gre options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.3"} Port "gre-ac100d07" Interface "gre-ac100d07" type: gre options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.7"} Port "gre-ac100d08" Interface "gre-ac100d08" type: gre options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.8"} Port "gre-ac100d05" Interface "gre-ac100d05" type: gre options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.5"} Port "gre-ac100d06" Interface "gre-ac100d06" type: gre options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.6"} Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port "gre-ac100d04" Interface "gre-ac100d04" type: gre options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.4"} Port br-tun Interface br-tun type: internal Bridge br-int fail_mode: secure Port "qvo86bf0e09-3f" tag: 2 Interface "qvo86bf0e09-3f" Port "qvo73d3f4d9-72" tag: 2 Interface "qvo73d3f4d9-72" Port br-int Interface br-int type: internal Port "qvo7c5fbeca-ab" tag: 2 Interface "qvo7c5fbeca-ab" Port "qvo2364aa3b-98" tag: 4 Interface "qvo2364aa3b-98" Port "qvo0194c615-74" tag: 5 Interface "qvo0194c615-74" Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port "qvo4a30d4d8-e3" tag: 3 Interface "qvo4a30d4d8-e3" Port "qvo1d912409-ed" tag: 2 Interface "qvo1d912409-ed" Port "qvoe79c563d-50" tag: 1 Interface "qvoe79c563d-50" Port "qvo4b146bb0-52" tag: 3 Interface "qvo4b146bb0-52" ovs_version: "2.5.0"
全部节点上链接bdc8的port同样,全都须要删除
# ovs-vsctl del-port br-tun gre-ac100d08