Recently I started to get a strange error with one of my OpenVZ VE nodes. This VE is the one that maintains my MySQL database for my WordPress blog. Here’s the error I was getting.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | # inside the OpenVZ VE % yum update Loading "fastestmirror" plugin Loading mirror speeds from cached hostfile * atrpms: dl.atrpms.net * extras: mirror.anl.gov * rpmforge: ftp-stud.fht-esslingen.de Traceback (most recent call last): File "/usr/bin/yum", line 29, in ? yummain.main(sys.argv[1:]) File "/usr/share/yum-cli/yummain.py", line 105, in main result, resultmsgs = base.doCommands() File "/usr/share/yum-cli/cli.py", line 289, in doCommands self._getTs() File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 85, in _getTs self._getTsInfo() File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 91, in _getTsInfo self._tsInfo.setDatabases(self.rpmdb, self.pkgSack) File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 537, in <lambda> pkgSack = property(fget=lambda self: self._getSacks(), File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 392, in _getSacks self.repos.populateSack(which=repos) File "/usr/lib/python2.4/site-packages/yum/repos.py", line 214, in populateSack self.doSetup() File "/usr/lib/python2.4/site-packages/yum/repos.py", line 66, in doSetup self.ayum.plugins.run('postreposetup') File "/usr/lib/python2.4/site-packages/yum/plugins.py", line 169, in run func(conduitcls(self, self.base, conf, **kwargs)) File "/usr/lib/yum-plugins/fastestmirror.py", line 90, in postreposetup_hook repomirrors[str(repo)] = FastestMirror(repo.urls).get_mirrorlist() File "/usr/lib/yum-plugins/fastestmirror.py", line 142, in get_mirrorlist self._poll_mirrors() File "/usr/lib/yum-plugins/fastestmirror.py", line 155, in _poll_mirrors pollThread.start() File "/usr/lib/python2.4/threading.py", line 416, in start _start_new_thread(self.__bootstrap, ()) thread.error: can't start new thread |
Yup, it made a lot of sense to me as well. Seemed to be something with yum’s fastermirror plugin, or so I thought, so I just disabled it and was able to get a little further like this:
1 2 3 | # inside the OpenVZ VE yum --disableplugin=fastermirror update |
However now the update would go through the motions of downloading packages etc. but would just die when trying to install them. So after googling for a bit I found a bug submitted against yum reporting a similar problem. Turns out this is yum and/or python’s way of saying it ran out of memory. So obvious, right?
So I checked my VE’s beancounters and low and behold it was running out of memory. You can tell when you start seeing a count in the failcnt column below on the privvmpages row. BTW, this VE instance is using the default value of 256MB of RAM, aka. 65536 pages * 4096KB = 268,435,456 bytes.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | # inside the OpenVZ VE % cat /proc/user_beancounters Version: 2.5 uid resource held maxheld barrier limit failcnt 104: kmemsize 2406222 2873722 14372700 14790164 0 lockedpages 0 0 256 256 0 privvmpages 45641 65547 65536 69632 53 shmpages 1 673 21504 21504 0 dummy 0 0 0 0 0 numproc 39 46 240 240 0 physpages 6694 20451 0 9223372036854775807 0 vmguarpages 0 0 33792 9223372036854775807 0 oomguarpages 6694 20451 26112 9223372036854775807 0 numtcpsock 7 23 360 360 0 numflock 9 13 188 206 0 numpty 1 1 16 16 0 numsiginfo 1 4 256 256 0 tcpsndbuf 122304 838656 1720320 2703360 0 tcprcvbuf 114688 860544 1720320 2703360 0 othersockbuf 18560 84768 1126080 2097152 0 dgramrcvbuf 0 13376 262144 262144 0 numothersock 17 24 120 120 0 dcachesize 0 0 3409920 3624960 0 numfile 865 1179 9312 9312 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 numiptent 10 10 128 128 0 |
Using the command below increased the VE’s memory from the default 256MB to 410MB. As an aside, I’ve never seen it mentioned on any blog/wiki/forum whatever, but you can specify the value to the –privvmpages arguement as either # pages, Gigabytes, Megabytes, or Kilobytes. I find this easier to use these than trying to calculate out the number of pages. In fact the only mention I’ve seen of these easier ways to specify them RAM is in vzctl’s man page. Guess it pays to read the man pages?
1 2 3 | # outside the OpenVZ VE, aka. on the OpenVZ Host Node vzctl set 104 --privvmpages 410m:450m --save |
Here is a snippet from the vzctl man page.
... You can also specify different suffixes for set parameters (except for the parameters which names start with num). For example, vzctl set CTID --privvmpages 5M:6M should set privvmpages' barrier to 5 megabytes and its limit to 6 megabytes. Available suffixes are: g, G -- gigabytes. m, M -- megabytes. k, K -- kilobytes. p, P -- pages (page is 4096 bytes on x86 architecture, other architectures may differ). You can also specify the literal word unlimited in place of a number. In that case the corresponding value will be set to LONG_MAX, i. e. the maximum possible value. ...
So now the VE instance has 104960 pages available aka. ~410MB, with a barrier limit of ~450MB, aka. 115200 pages, and yum is happy.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | # inside the OpenVZ VE % cat /proc/user_beancounters Version: 2.5 uid resource held maxheld barrier limit failcnt 104: kmemsize 2431123 2891301 14372700 14790164 0 lockedpages 0 0 256 256 0 privvmpages 46320 74647 104960 115200 54 shmpages 1 673 21504 21504 0 dummy 0 0 0 0 0 numproc 39 49 240 240 0 physpages 7107 20451 0 9223372036854775807 0 vmguarpages 0 0 33792 9223372036854775807 0 oomguarpages 7107 20451 26112 9223372036854775807 0 numtcpsock 7 23 360 360 0 numflock 9 13 188 206 0 numpty 1 1 16 16 0 numsiginfo 1 4 256 256 0 tcpsndbuf 122304 838656 1720320 2703360 0 tcprcvbuf 114688 860544 1720320 2703360 0 othersockbuf 20880 84768 1126080 2097152 0 dgramrcvbuf 0 13376 262144 262144 0 numothersock 17 24 120 120 0 dcachesize 0 0 3409920 3624960 0 numfile 873 1179 9312 9312 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 numiptent 10 10 128 128 0 |
March 24th, 2009, @ 11:31 pm |
Tags: 
Category: 
