View previous topic :: View next topic |
Author |
Message |
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Sat Aug 23, 2014 7:57 pm Post subject: HOWTO: A modern init system in 3 minutes |
|
|
Today we're going to install runit. Some good reasons to switch:
Code: | # ls -l /sbin/*init
-rwxr-xr-x 1 root root 35308 Mar 8 21:43 /sbin/init
-rwxr-xr-x 1 root root 13952 Aug 23 20:19 /sbin/runit-init
# du -sh /usr/share/doc/runit*
200K /usr/share/doc/runit-2.1.1-r1/ |
Install it:
Code: | # emerge -av runit
# $EDITOR /etc/runit/1 |
Make sure to boot into a useful runlevel:
/etc/runit/1: | ...
RUNLEVEL=S /sbin/rc sysinit
RUNLEVEL=S /sbin/rc boot
/sbin/rc default # <== add this line
... |
The scripts inside /etc/runit/runsvdir are configured by default to run agetty on tty1-6, like sysvinit's inittab does. Be careful about replacing these; I found mingetty to not work.
Edit your bootloader stuff to append "init=/sbin/runit-init" to the command line.
Then reboot:
|
|
Back to top |
|
|
wuzzerd Guru
Joined: 05 Jan 2005 Posts: 467 Location: New Mexico
|
Posted: Sun Aug 24, 2014 12:46 am Post subject: |
|
|
Very interesting, Ant. It seems to run out of the box. One snag with the parallel starting of services: ntp-client fails
with server not found messages. I moved it to /etc.local.d with a bit of sleep.
<troll> It is much easier to set up and debug than systemd </troll> |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Sun Aug 24, 2014 9:28 pm Post subject: |
|
|
A few other things I've noticed: - There is no shutdown or halt command, and trying to run the sysvinit ones will only complain that /dev/initctl doesn't exist. Shutdown/reboot is done by using runit-init(8).
- Many tools assume those sysvinit commands exist (I wonder if they're equally broken under systemd), and will need reconfiguring (or replacing with less broken ones if they don't allow that).
|
|
Back to top |
|
|
Dr.Willy Guru
Joined: 15 Jul 2007 Posts: 547 Location: NRW, Germany
|
Posted: Mon Aug 25, 2014 10:35 pm Post subject: Re: HOWTO: A modern init system in 3 minutes |
|
|
Hi Ant P.
Brilliant find. I'm still having a look at it, but doesn't this…
Ant P. wrote: | /etc/runit/1: | ...
RUNLEVEL=S /sbin/rc sysinit
RUNLEVEL=S /sbin/rc boot
/sbin/rc default # <== add this line
... |
|
…kindof defeat the purpose of installing it in the first place?
I mean in this setup you're just using it as an openrc wrapper.
Edit:
Ok, so …
Code: | runit-+-acpid
|-dwmstatus
|-gpg-agent
|-runsvdir-+-4*[runsv---agetty]
| |-runsv---xlshd---Xorg---2*[{Xorg}]
| |-runsv---dhcpcd
| |-runsv---mpd---4*[{mpd}]
| |-runsv---vsftpd
| `-runsv---syslog-ng
|-systemd-udevd
|-tmux-+-zsh-+-pstree
| | `-xclip
| |-zsh---htop
| `-zsh
|-wpa_supplicant
`-xlsh---dwm-+-firefox---35*[{firefox}]
`-xterm---su---zsh---tmux
|
I just converted my entire "default" runlevel to runit.
I wonder if and when I will run into problems, because right now I'm liking it: Setup was easy and the cli is pretty nice too. |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Tue Aug 26, 2014 12:30 am Post subject: |
|
|
Yes, you have a point, it seems pointless at first glance.
The major reason why you'd want runit over sysv is its own process supervision tools, which are good for custom services. I use it for a few game servers and some out-of-tree/self-written programs (things that need to stay running even if they segfault periodically). Most users will only ever care about the stuff given to them in /etc/init.d, and that's fine too. There's nothing wrong with keeping OpenRC (or even anything else that originally ran atop sysvinit...)
One other reason is most of runit's tools exist in busybox, so there's the possibility of using this for doing a tiny install.
But more importantly, without going off on a long tangent about UNIX philosophy, the simple ability to do things like this without having to deal with a flag day demonstrates the system is working correctly. |
|
Back to top |
|
|
Dr.Willy Guru
Joined: 15 Jul 2007 Posts: 547 Location: NRW, Germany
|
Posted: Wed Aug 27, 2014 11:10 pm Post subject: |
|
|
Ok, so I searched a bit deeper and found out that runit implements a concept by djb (whom you might know) called daemontools.
Now there are two things I was asking myself:
1) There are other programs implementing that concept, most notably (in my opinion) s6.
I find it really interesting and think it might be worthwhile to migrate to it entirely, replacing openrc. On the other hand I've only had a quick look and don't know a lot about init systems in the first place.
So I'd like to hear what other people think of it (s6 and the daemontools concept in general)
2) As a follow up to 1) Do you think it would be viable/worthwhile to implement the services required to bring up a gentoo system? |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Sun Aug 31, 2014 12:30 pm Post subject: |
|
|
I've been looking at s6 too, I don't think I'm ready to go *that* far yet (execline's syntax feels like a cross between sh and Haskell). I haven't had much luck installing it either.
Regarding the second thing, I'm too paranoid to touch anything in OpenRC's boot runlevel (that's probably why runit's configured to do it by default). Apart from that I've been making good progress. On my netbook I've ported over all the simple daemons in the default runlevel (basically anything that fits on one screen), and now it looks like this:
pstree: | runit─┬─cron
├─metalog───metalog
├─nginx───nginx
├─ntpd
├─runsvdir─┬─3*[runsv───agetty]
│ ├─runsv───bluetoothd
│ ├─runsv───acpid
│ ├─runsv───avahi-daemon───avahi-daemon
│ ├─runsv───dbus-daemon
│ ├─runsv─┬─connmand
│ │ └─svlogd
│ └─runsv─┬─svlogd
│ └─wpa_supplicant
├─slim───X
├─sshd───sshd───sshd─┬─bash───pstree
│ └─4*[{sshd}]
└─udevd
|
With rc_parallel=YES, I get to an X login prompt in 10.5 seconds. It could probably be quicker, I have net.lo eating half a second of CPU just to "provide net" to other init.d scripts. OpenRC needs some equivalent of package.provided...
After using runit for a few days I've encountered several other problems, which I'm not clear on how to fix: - SysV seems to do some internal magic with VCs that runit doesn't. The getty-tty* services end up spawning bash processes with no job control, which means Ctrl+C doesn't work in them, among other things. I suspect this is the same root cause that's making mingetty misbehave. (I would've tried kmscon, but it fails to build.)
- The shutdown (/etc/runit/3) script supplied by Gentoo has syntax errors (sv shutdown -w instead of sv -w shutdown). Correcting those errors causes it to wait 5 minutes on any logged in virtual consoles to exit.
|
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Mon Sep 01, 2014 4:56 pm Post subject: |
|
|
I wrote this to make sysv's poweroff/reboot programs work. You can just run it directly.
/etc/runit/runsvdir/all/initctl/run: | #!/usr/bin/perl -T
use v5.18;
use warnings;
use autodie;
$ENV{PATH} = '/sbin:/bin';
my $INIT_FIFO = '/dev/initctl';
my $REQ_SIZE = 384;
my $MAGIC_HEADER = 0x03091969;
my $CMD_RUNLEVEL = 0x1;
-e $INIT_FIFO or system("/usr/bin/mkfifo --mode=0600 $INIT_FIFO");
open(my $pipefh, q{<}, $INIT_FIFO);
while () {
read($pipefh, my $request, $REQ_SIZE) == $REQ_SIZE
or die "Failed to read $REQ_SIZE bytes";
my ($magic, $command, $runlevel, $sleeptime, $stuff) = unpack('l l A4 l A*', $request);
$magic == $MAGIC_HEADER or die "Got invalid magic bytes";
$command == $CMD_RUNLEVEL or next;
$runlevel =~ /^(0|6)$/ and system("/sbin/runit-init $1");
} |
(Side note: sysv's source code is... "interesting". No wonder everyone just starts from scratch instead of fixing it...) |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Fri Sep 05, 2014 2:31 pm Post subject: |
|
|
I've solved the getty/console weird behaviour - it needs to run in its own process group, i.e.:
/etc/runit/runsvdir/all/getty-tty1: | #!/bin/sh
exec /usr/bin/chpst -P /sbin/agetty 38400 tty1 linux
|
(this also fixes mingetty) |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Sat Sep 27, 2014 6:21 pm Post subject: |
|
|
Follow-up to previous post: it turns out the runit sources already contain a /etc/runit/2 file which uses "runsvdir -P". However Gentoo installs its own incorrect version of that, causing the breakage.
Using that file would be more correct than the fix above, and is what I've fixed the runit ebuild to do in my local overlay.
I have had no response to any of the runit-related bugs I've filed on b.g.o in the last month, FWIW. |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Sat Sep 27, 2014 7:00 pm Post subject: |
|
|
Ant P. wrote: | Follow-up to previous post: it turns out the runit sources already contain a /etc/runit/2 file which uses "runsvdir -P". However Gentoo installs its own incorrect version of that, causing the breakage.
Using that file would be more correct than the fix above, and is what I've fixed the runit ebuild to do in my local overlay. |
That makes sense; have you posted the patch to the ebuild to b.g.o? Link (or patch) please, so it's all in one place. |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Sat Sep 27, 2014 7:32 pm Post subject: |
|
|
Thanks for showing interest.
My ebuild's diverged too much at this point to get a clean diff, but the change is simple enough — fix the path in "${S}"/../etc/2 (/service -> /var/service), install that file instead of "${FILESDIR}"/2, and then the latter can be removed from the tree.
For reference, I'll start keeping track of all the runit bugs I've filed here:
|
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Sat Sep 27, 2014 8:04 pm Post subject: |
|
|
522204 (login shells are broken - patch obsoleted by above fix)
You should point that out on the bug then, by obsoleting that patch in favour of a minimal patch to the ebuild, as you describe, with comment about no longer needing one of the files etc.
The shutdown (sv -w) one looks like it's installing its own file too; is there good reason for that, or would upstream's be better? |
|
Back to top |
|
|
Shamus397 Apprentice
Joined: 03 Apr 2005 Posts: 218 Location: Ur-th
|
Posted: Mon Sep 29, 2014 2:10 pm Post subject: |
|
|
Just wanted to chime in and say brilliant work Ant P., I'll be watching this thread with interest. |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Mon Sep 29, 2014 4:21 pm Post subject: |
|
|
I'll get around to it when I see any indication that the maintainer is even trying. Other people on the forums seem to have an extremely low opinion of him, so I decided it's better to direct my efforts toward a fixed ebuild people can use *now*.
Quote: | The shutdown (sv -w) one looks like it's installing its own file too; is there good reason for that, or would upstream's be better? |
The 1 & 3 files are Gentoo-specific, or more accurately, OpenRC-specific. We can't get rid of those, upstream has versions for a few different distros but not ours. |
|
Back to top |
|
|
depontius Advocate
Joined: 05 May 2004 Posts: 3523
|
Posted: Mon Sep 29, 2014 4:51 pm Post subject: |
|
|
Ant P. wrote: |
I'll get around to it when I see any indication that the maintainer is even trying. Other people on the forums seem to have an extremely low opinion of him, so I decided it's better to direct my efforts toward a fixed ebuild people can use *now*.
Quote: | The shutdown (sv -w) one looks like it's installing its own file too; is there good reason for that, or would upstream's be better? |
The 1 & 3 files are Gentoo-specific, or more accurately, OpenRC-specific. We can't get rid of those, upstream has versions for a few different distros but not ours. |
There has been any amount of discussion about forking Gentoo. But an incremental point would be forking OpenRC, but that would also mean someone signing up to maintain the fork, and presumably an overlay to hold it. At the moment I don't have the time to do that, so I'm not going to presume to sign someone else up, either. I suspect I also don't currently have the proper skills refined, but that's a correctable situation. _________________ .sigs waste space and bandwidth |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Mon Sep 29, 2014 6:00 pm Post subject: |
|
|
Ant P. wrote: | I'll get around to it when I see any indication that the maintainer is even trying. Other people on the forums seem to have an extremely low opinion of him, so I decided it's better to direct my efforts toward a fixed ebuild people can use *now*. |
Hmm; can't say I agree with this approach. A distro is bigger than any one person, and where are those people supposed to find the ebuild, when they search bugzilla for this problem?
Just ignore the person, and speak to your fellow users, including over bugzie: Gentoo is first and foremost about its users, whatever anyone might try to claim. So is every other distro out there.
Quote: | The 1 & 3 files are Gentoo-specific, or more accurately, OpenRC-specific. We can't get rid of those, upstream has versions for a few different distros but not ours. |
Fair enough; we can do initscripts. I don't like the names though. Too easy for a typoed script to overwrite. |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Mon Sep 29, 2014 6:01 pm Post subject: |
|
|
depontius wrote: | There has been any amount of discussion about forking Gentoo. But an incremental point would be forking OpenRC, but that would also mean someone signing up to maintain the fork, and presumably an overlay to hold it. At the moment I don't have the time to do that, so I'm not going to presume to sign someone else up, either. I suspect I also don't currently have the proper skills refined, but that's a correctable situation. |
Please don't sign up in the hope that you can "refine" your skillset on the job. We've already had one incompetent mangle the project. |
|
Back to top |
|
|
depontius Advocate
Joined: 05 May 2004 Posts: 3523
|
Posted: Tue Sep 30, 2014 1:09 am Post subject: |
|
|
steveL wrote: | depontius wrote: | There has been any amount of discussion about forking Gentoo. But an incremental point would be forking OpenRC, but that would also mean someone signing up to maintain the fork, and presumably an overlay to hold it. At the moment I don't have the time to do that, so I'm not going to presume to sign someone else up, either. I suspect I also don't currently have the proper skills refined, but that's a correctable situation. |
Please don't sign up in the hope that you can "refine" your skillset on the job. We've already had one incompetent mangle the project. |
I know better than to even try such a thing. That's right up there with delivering something on the Friday before you leave for a 2-week vacation. If I were so inclined I'd be refining my applicable skillset before even thinking of taking on the job.
I don't mean to sign you up, but you've been the one talking about forking Gentoo. Gentoo doesn't need forking, OpenRC does. Given the situation where I work, I know benign and/or non-benign neglect when I see it. The lead point of weakness in the systemd advance on Gentoo is OpenRC, which you appear to have said is in a state of under or negative development. Perhaps you're a natural focal point on a fork of OpenRC, at the very least a more modest project than forking all of Gentoo. Incidentally, I'm running "ebuild install" against openrc-0.13.1 now, just to see what is presumably in the future. The Changelog wasn't very informative, something about a regression and something about "removal". It sits beside the sysvinit that I've done the same with, and occasionally peruse the source for.
EDIT - I'll test and give feedback. _________________ .sigs waste space and bandwidth |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Tue Sep 30, 2014 4:48 am Post subject: |
|
|
depontius wrote: | I know better than to even try such a thing. That's right up there with delivering something on the Friday before you leave for a 2-week vacation. If I were so inclined I'd be refining my applicable skillset before even thinking of taking on the job. |
Phew; glad you're as sane as I thought.
Quote: | I don't mean to sign you up, but you've been the one talking about forking Gentoo. |
Oh blimey, I have always argued against any such attempt, and have consistently said there's no point forking Gentoo ("try an overlay"). As the whole point of Gentoo is that everything is from source, and I've never met anyone who wanted to do more than patch a small corner. Claims to want to do much more, sure, but never the reality, imo.
Just to get that straight; yes I mentioned the idea, but only to dismiss it immediately. I most definitely don't want to fork Gentoo.
Quote: | Gentoo doesn't need forking, OpenRC does. Given the situation where I work, I know benign and/or non-benign neglect when I see it. The lead point of weakness in the systemd advance on Gentoo is OpenRC, which you appear to have said is in a state of under or negative development. Perhaps you're a natural focal point on a fork of OpenRC, at the very least a more modest project than forking all of Gentoo. Incidentally, I'm running "ebuild install" against openrc-0.13.1 now, just to see what is presumably in the future. The Changelog wasn't very informative, something about a regression and something about "removal". It sits beside the sysvinit that I've done the same with, and occasionally peruse the source for.
EDIT - I'll test and give feedback. |
Hmm well I'd be up for going back and filtering out the crappy shell that's infected the project since Roy left; I find it quite offensive.
blueness's work on eudev (filtering commits from udev for sanity) showed the way. So that'd be going back and reviewing each commit, rewriting it to do whatever the patch was supposed to do, without losing robust POSIX sh. So tedious, but not intellectually hard, just time-consuming.
If you really want to fork properly, I'd have to get help on the C side from my boss, most likely. We've discussed it before now, in the context of how little progress had happened with the actual code (the only significant progress made has been by users, who've not been encouraged from what I've seen.) While he'd be into helping, I'm sure, he also wants to sort out the mactab idea in-kernel, and he's not a kernel coder, bless ;) so we'd need a couple of more people at least, I think.
And someone like you would have to drive the project; the lead is often more of a QA admin than deep-tech, ime, since herding cats is a different skillset to holding a program in your head. It's very rare for the two sets of abilities to combine (zmedico is the only one I've met personally, apart from people in previous generations), but it's important that both sides know their limitations, or you end up with bitchy fallouts and no progress.
I know for sure I'd be a terrible lead; I can help on the technical side, but I simply don't have the demeanour, nor the tact, required for that.
I won't be able to even look at it for at least three weeks, realistically; my free-code time is going on update atm, and then the installer, which hopefully won't require much to do what it did before. That'll be useful for testing kde-lean, which is also waiting on me to finish update --toolchain. Surgery was necessitated by issues brought up by perl, ie a toolchain package in /etc/warning, which could be, and was ofc, blocked. From a user pov it's still at the same state (in git) wrt perl; in the background lots of changes have happened, which will be used by the next part I'm working on now.
Let me know how you get on with the newer version of openrc, and how you feel after you've poked through the codebase. (Reviewing the history with gitview is very revealing.) |
|
Back to top |
|
|
depontius Advocate
Joined: 05 May 2004 Posts: 3523
|
Posted: Tue Sep 30, 2014 11:39 am Post subject: |
|
|
steveL wrote: |
Let me know how you get on with the newer version of openrc, and how you feel after you've poked through the codebase. (Reviewing the history with gitview is very revealing.) |
I haven't actually installed the new OpenRC - I did an "ebuild install", so it's sitting out in /var/tmp/portage, along with sysvinit and the stable openrc. It's a good way to peruse the code - the source is there as well as an installation image. You can see both gozintas and gozoutas for the build process.
Thanks for the pointer to gitview - I see now that I have it installed on one system. I've been trying to learn more about Android so I can help out with my languishing phone, and don't know much about git. Sometimes the problem with learning curves is getting a good handhold on the bottom of the ladder. _________________ .sigs waste space and bandwidth |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Tue Sep 30, 2014 7:58 pm Post subject: |
|
|
depontius wrote: | Thanks for the pointer to gitview - I see now that I have it installed on one system. |
Yeah it comes with git on Gentoo systems; to get the git-gui and gitk you need the tk USE-flag on the git package.
I just found that out the other day; until then I'd been wondering what had happened to gitk (and why i didn't have the git-gui command) for a few years lul. I just kept using gitview, and now I'm settled in with it as an overview on history. Oh, gitview --all is likely better for reviewing a repo.
Quote: | I've been trying to learn more about Android so I can help out with my languishing phone, and don't know much about git. Sometimes the problem with learning curves is getting a good handhold on the bottom of the ladder. |
Yeah; git is quite complex as well. Try this url; I wish I'd seen it when I was starting out.
The main thing to remember is that the "index" is better thought of as the "stage"; it's what you've staged to make a commit (and you can add to it, or back things out with reset -p, all without committing). The other important thing is that a commit is only ever local; nothing gets sent until you push, typically several commits together. So you can always correct yourself, up until the time you git push.
I'd advise you not to use git-rebase, ever, except for a branch that you're rebasing onto master/upstream (don't worry about it til you need it.) The best command I've found is git add -p somefile, which allows you to select which bits of the file will go into the commit. (ie the staged index.) That covers the main usage people have for rebase -i, at least in pragmatic terms: being able to break up a commit into logical parts, and then push several commits all at once, each one being distinct, but all of them being required for the thing to be considered working. (which is where the distinction between commit and push is vital.)
But you keep the forward sense of progress. That's what always trips me up with rebase (the feeling of going backwards), and rebase can bork your repo, so I just avoid it now.
Check this post out too; it's my config and aliases for git, which should give you an idea of how i use it day-to-day; not very expertly compared to many but I've come to vcs very late in my life, and I try to keep the workflow simple, so I have enough headspace for coding. (Hence the insistence on fast-forward merges, unless I explicitly ask for a fork.)
Whenever I'm stuck or a bit unsure despite the manpages, I just /join #git -- they're brilliant :-)
Although they do have a tendency to make everything about the git model, and explaining all the concepts ;)
I really do recommend the kdiff3 setup (or some other difftool) as per the linked post(vdiff somefile), as well. It's so nice being able to view the patches in the context of the whole file, from time to time; again an alternative view that helps keep perspective on the overall process.
I use it when I'm coming back to work, to get my head back into the code, and quite often before I commit; it makes for much better self-review, ime, simply because you have the whole file to look at, and can move through it very quickly to where the changes are, but still consider other options based on everything else it fits into.
gdiff/git diff (the standard util ofc) is more useful pre-commit when I'm already in the flow, since I'm just reviewing the detail; but having the option to step back with kdiff3 really helps, ime.
Oh and don't forget git diff --staged (staged/vstaged aliases) to review what you're about to commit. You can always commit --amend it after as well, so don't be shy; it's not til you git push that anything leaves your machine, and you can always reset HEAD^ to go back a step (add --soft to keep your staged changes in the index).
Like I said though, the tias link is what you need, since it gets you playing locally, and takes the fear out of the process; at this point I'd advise you to play with branches and checkouts as well, in a throwaway repo (vbranch/git branch -lvv to see what's going on.) In this instance a push simply goes to the other local repo, so again you can't harm anything, since it's just another throwaway directory.
I'm sure you've been swamped with other links already; if not feel free to /msg friendlyToaster git on freenode.
And /join #git if you want advice; we're in #friendly-coders btw (i'm 'igli'), in case it all gets too much ;-) |
|
Back to top |
|
|
bnolsen n00b
Joined: 07 Oct 2014 Posts: 1
|
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Thu Oct 09, 2014 8:51 pm Post subject: |
|
|
Okay, one of my above posts turned out to be incorrect. runsvdir -P doesn't exactly do the same thing as chpst -P, and I ended up with a few wedged boxes today that I couldn't log into after a reboot, because mingetty wouldn't start at all. Whoops.
(I guess I don't have to update that bug after all ) |
|
Back to top |
|
|
steveL Watchman
Joined: 13 Sep 2006 Posts: 5153 Location: The Peanut Gallery
|
Posted: Fri Oct 10, 2014 12:32 am Post subject: |
|
|
Ant P. wrote: | Whoops.
(I guess I don't have to update that bug after all :lol: ) |
lul :) |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|