Keeping CentOS 5 OpenVZ images up to Date with Yum

Now that I’ve been using OpenVZ for several months I’d gotten to the point where I wanted/needed to “yum update” all my VEs. I currently have 11 images running on my OpenVZ Server. I thought I could just vzctl exec … yum -y update all the VEs, but quickly ran into some issues with this brute force approach. Doing the yum -y update broke several of my VEs so I opted to restore the unrecoverable ones from backups.

For my second attempt, I opted to do each VE independently, to get a better understanding of what the best approach would be for doing mass upgrades like this now, and in the future. The first hurdle to overcome had to do with some of the VEs running out of memory (RAM) during the upgrade process.

NOTE: A VE = Virtual Environment (aka. a virtual host), while HN = Host Node

Issue #1, not enough RAM for yum to run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# RAM budgets for VEs
 
# NOTE: values are in # of pages (1 pg. = 4K)
 
% (printf "vm feature held maxheld barrier limit failcnt\n"; grep privvm /proc/bc/1*/resources)|column -t
vm                       feature      held    maxheld  barrier  limit   failcnt
/proc/bc/101/resources:  privvmpages  31904   82215    65536    69632   2
/proc/bc/102/resources:  privvmpages  119803  196620   166400   179200  9517
/proc/bc/103/resources:  privvmpages  27125   35974    65536    69632   0
/proc/bc/104/resources:  privvmpages  56251   107250   104960   115200  0
/proc/bc/105/resources:  privvmpages  73559   82926    98304    103304  0
/proc/bc/106/resources:  privvmpages  30219   68097    65536    69632   0
/proc/bc/108/resources:  privvmpages  30081   84291    65536    69632   1
/proc/bc/109/resources:  privvmpages  32790   74199    98304    103304  0
/proc/bc/110/resources:  privvmpages  40497   69408    65536    69632   1
/proc/bc/111/resources:  privvmpages  26990   35371    65536    67840   0
 
# NOTE: converted the columns to megabytes (MB), it's just easier to read
 
(printf "vm feature held maxheld barrier limit failcnt\n"; grep privvm /proc/bc/1*/resources|awk '{sub($3,$3*4096/2^20) sub($4,$4*4096/2^20) sub($5,$5*4096/2^20) sub($6,$6*4096/2^20)}1')|column -t
vm                       feature      held     maxheld  barrier  limit    failcnt
/proc/bc/101/resources:  privvmpages  124.625  321.152  256      272      2
/proc/bc/102/resources:  privvmpages  467.98   768.047  650      700      9517
/proc/bc/103/resources:  privvmpages  105.957  140.523  256      272      0
/proc/bc/104/resources:  privvmpages  219.73   418.945  410      450      0
/proc/bc/105/resources:  privvmpages  287.34   323.93   384      403.531  0
/proc/bc/106/resources:  privvmpages  118.043  266.004  256      272      0
/proc/bc/108/resources:  privvmpages  117.504  329.262  256      272      1
/proc/bc/109/resources:  privvmpages  128.086  289.84   384      403.531  0
/proc/bc/110/resources:  privvmpages  158.191  271.125  256      272      1
/proc/bc/111/resources:  privvmpages  105.43   138.168  256      265      0

According to this data 4 of the 11 images had gone over their allocation of RAM. So I tried restarting these and re-running yum update within the problem VEs. Again the update failed and so I needed to increase their allocation of memory. I didn’t want to devote more memory permanently, just a enough temporarily to do the upgrade. So I used this trick to temporarily bump up a VEs allocated memory.

1
2
3
4
5
6
7
8
9

# increase the RAM by 100MB
vzctl set 101 --privvmpages $((256+100))m:$((272+100))m --save
 
# ...
# do the upgrade (yum update)
# ...
 
# decrease the RAM back to the original value
vzctl set 101 --privvmpages 256m:272m --save

For the remaining 3 VEs that needed additional memory I used these commands to increase their allocations of RAM

1
2
3
4
5
6
7
8
9

# cmds. to increase RAM by 100MB
vzctl set 102 --privvmpages $((650+100))m:$((700+100))m --save
vzctl set 108 --privvmpages $((256+100))m:$((272+100))m --save
vzctl set 110 --privvmpages $((256+100))m:$((272+100))m --save
 
# cmds. to decrease
vzctl set 102 --privvmpages 650m:700m --save
vzctl set 108 --privvmpages 256m:272m --save
vzctl set 110 --privvmpages 256m:272m --save

Issue #2, not enough diskspace for yum to run

The next snag I ran into had to do with a couple of the VEs running out of diskspace. And here’s the commands I used to reconfigure more diskspace.

1
2
3
4

# reconfigure with 2GB of diskspace
vzctl set 101 --diskspace 2048M:2252M --save
vzctl set 102 --diskspace 2048M:2252M --save
vzctl set 103 --diskspace 2048M:2252M --save

At this point I was able to yum update all my images, and work through any packages that had been updated.

Issue #3, unfinished yum transactions

One final snag that I didn’t really pick up on until probably the 2nd or 3rd time I went through this “process” was that some of the VEs would report during yum update that there were incomplete transactions, and that I really should run the yum-complete-transaction command. I’d never heard of this command, so I went looking and apparently if yum fails to complete a transaction, it remains queued up, causing you to get warnings during subsequent yum updates. So to get yum-complete-transaction command you need to install a package called yum-utils.

1

yum install yum-utils

Then you’ll want to run yum-complete-transaction. Pay special attention to how many unfinished transactions this command finds. Multiple yum transactions can get queued up so you may need to run yum-complete-transaction multiple times. If you do need to run it more than once, it runs the last unfinished transaction first, and then the 2nd to last, and so on until they’re all completed. Don’t be alarmed if it says it’s removing what appear to be critical packages such as bash. This occured on several of my systems and it turned out that there was multiple bashes installed (at least according to the RPM database) and the unfinished transaction was merely trying to uninstall the previous version, leaving the latest version intact.

A bonus to running yum-complete-transaction, is that it helps reduce the memory footprint that yum needs when it has unfinished transactions vs. when it doesn’t. I was seeing yum using 50-100MB in some cases due to all the unfinished transactions, so fixing this issue helped with minimizing future problems that I may see related to issue #1, not enough RAM.

Final Thoughts

My original plan was that I would create a single OpenVZ image that included everything that I needed as far as packages go, and then selectively turn on and off apps that I didn’t need, like apache, for example. Going forward I’m going to try a completely different approach, and thin out each image to only include the bare minimum, to perform it’s job, i.e. mail server, samba server, etc. This should help out with a lot of redundant updating to all the VEs with packages that only 1 or 2 will actually need. The downside with this approach, and reason I avoided it to begin with, was that it seemed to be more work in maintaining many different images, but in practice this may not be as bad as having a single monolithic image multiplied many times across many VEs.

OpenVZ is still a tremendous tool here. Many times when I ran into a problem, I simply restored an image from backup and then resumed, so this isn’t a slam on it, and more a workflow issue and what’s the best way to manage many instances of machines. Really what I want is a way to manage roles of machines. Perhaps it’s time to revisit using something like puppet?