Our inventory has two kinds of group. Role groups say what a server does: web, reporting, batch. Site groups say where it is: site_croydon, site_leeds. Both kinds sit directly under all. Site variables are mostly network things, the local DNS and NTP servers, but when we built Leeds as a home for small utility VMs somebody also put a sensible default there:
jvm_heap: 2g
The reporting group has its own value, because month end reports load a lot into memory:
jvm_heap: 12g
In September we moved the three reporting servers from Croydon to Leeds, which in the inventory meant moving them from one site group to another. The playbook ran clean. On the first working day of October every large month end report failed with OutOfMemoryError, and the JVM command line on each server said -Xmx2g.
When a host belongs to two groups, Ansible merges their variables in an order. A child group overrides its parent. Groups at the same level, which ours were, are merged alphabetically, and the group loaded last wins.…