Upscaling OpenShift application nodes (#571)

* scale-up: playbook for upscaling app nodes

* scale-up: removed debug

* scale-up: made suggested changes

* scale-up: indentation fix

* upscaling: process split into two playbooks that are executed by a bash script

- upscaling_run.sh: bash script, usage displayed using -h parameter
- upscaling_pre-tasks: check that new value is higher, change inventory variable
- upscaling_scale-up: rerun provisioning and installation, verify change

* upscaling_run: fixed openshift-ansible-contrib directory name

* upscaling_run: inventory can be entered as relative path

* upscaling_scale-up: fixed formatting

* upscaling: minor changes

* upscaling: moved to .../provisioning/openstack directory, README updated, minor changes made

* README: minor changes

* README: formatting

* uspcaling: minor fix

* upscaling: fix

* upscaling: added customisations, fixes

- openshift-ansible-contrib and openshift-ansible paths are customisable
- fixed implicit incrementation by 1

* upscaling: fixes

* upscaling: fixes

* upscaling: another fix

* upscaling: another fix

* upscaling: fix

* upscaling: back to a single playbook, README updated

* minor fix

* pre_tasks: added labels for autoscaling

* scale-up: fixes

* scale-up: fixed host variables, post-verification is only based on labels

* scale-up: added openshift-ansible path customisation

- path has to be absolute, cannot contain '/' at the end

* scale-up: fix

* scale-up: debug removed

* README: added docs on openshift_ansible_dir, note about bastion

* static_inventory: newly added nodes are added to new_nodes group

- note: re-running provisioning fails when trying to install docker

* removing new line

* scale-up: running byo/config.yml or scaleup.yml based on the situation

- (whether there is an existing deployment or not)

* openstack.yml: indentation fix

* added refresh inventory

* upscaling: new_nodes only contains new does, it is not used during the first deployment

* static_inventory: make sure that new nodes end up only in their new_nodes group

* bug fixes

* another fix

* fixed condition

* scale-up, static_inventory role: all app node data gathered before provisioning

* upscaling: bug fixes

* upscaling: another fixes

* fixes

* upscaling: fix

* upscaling: fix

* upscaling: another logic fix

* bug fix for non-scaling deployments

3 files changed, 43 insertions, 2 deletions

diff --git a/roles/static_inventory/tasks/filter_out_new_app_nodes.yaml b/roles/static_inventory/tasks/filter_out_new_app_nodes.yaml
new file mode 100644
index 000000000..826efe78d
--- /dev/null
+++ b/roles/static_inventory/tasks/filter_out_new_app_nodes.yaml
@@ -0,0 +1,15 @@
+---
+- name: Add all new app nodes to new_app_nodes
+  when:
+  - 'oc_old_app_nodes is defined'
+  - 'oc_old_app_nodes | list'
+  - 'node.name not in oc_old_app_nodes'
+  - 'node["metadata"]["sub-host-type"] == "app"'
+  register: result
+  set_fact:
+    new_app_nodes: '{{ new_app_nodes }} + [ {{ node }} ]'
+
+- name: If the node was added to new_nodes, remove it from registered nodes
+  set_fact:
+    registered_nodes: '{{ registered_nodes | difference([ node ]) }}'
+  when: 'not result | skipped'
diff --git a/roles/static_inventory/tasks/openstack.yml b/roles/static_inventory/tasks/openstack.yml
index e36974d93..adf78c966 100644
--- a/roles/static_inventory/tasks/openstack.yml
+++ b/roles/static_inventory/tasks/openstack.yml
@@ -37,7 +37,6 @@
       with_items: "{{ registered_nodes|difference(registered_nodes_floating) }}"
       add_host:
         name: '{{ item.name }}'
-        groups: '{{ item.metadata.group }}'
         ansible_host: >-
           {% if use_bastion|bool -%}
           {{ item.name }}
@@ -57,7 +56,6 @@
       with_items: "{{ registered_nodes_floating }}"
       add_host:
         name: '{{ item.name }}'
-        groups: '{{ item.metadata.group }}'
         ansible_host: >-
           {% if use_bastion|bool -%}
           {{ item.name }}
@@ -80,6 +78,30 @@
           {{ item.public_v4 }}
           {%- endif %}
 
+    # Split registered_nodes into old nodes and new app nodes
+    # Add new app nodes to new_nodes host group for upscaling
+    - name: Create new_app_nodes variable
+      set_fact:
+        new_app_nodes: []
+
+    - name: Filter new app nodes out of registered_nodes
+      include: filter_out_new_app_nodes.yaml
+      with_items: "{{ registered_nodes }}"
+      loop_control:
+        loop_var: node
+
+    - name: Add new app nodes to the new_nodes section (if a deployment already exists)
+      with_items: "{{ new_app_nodes }}"
+      add_host:
+        name: "{{ item.name }}"
+        groups: new_nodes, app
+
+    - name: Add the rest of cluster nodes to their corresponding groups
+      with_items: "{{ registered_nodes }}"
+      add_host:
+        name: '{{ item.name }}'
+        groups: '{{ item.metadata.group }}'
+
     - name: Add bastion node to inventory
       add_host:
         name: bastion
diff --git a/roles/static_inventory/templates/inventory.j2 b/roles/static_inventory/templates/inventory.j2
index 8863fb7c4..9dfbe3a5b 100644
--- a/roles/static_inventory/templates/inventory.j2
+++ b/roles/static_inventory/templates/inventory.j2
@@ -40,6 +40,7 @@ dns
 nodes
 etcd
 lb
+new_nodes
 
 # Set variables common for all OSEv3 hosts
 [OSEv3:vars]
@@ -78,6 +79,8 @@ dns.{{ stack_name }}
 [lb:children]
 lb.{{ stack_name }}
 
+[new_nodes:children]
+
 # Empty placeholders for all groups of the cluster nodes
 [masters.{{ stack_name }}]
 [etcd.{{ stack_name }}]
@@ -86,6 +89,7 @@ lb.{{ stack_name }}
 [app.{{ stack_name }}]
 [dns.{{ stack_name }}]
 [lb.{{ stack_name }}]
+[new_nodes.{{ stack_name }}]
 
 # BEGIN Autogenerated groups
 {% for group in groups %}