Tools for Group Administration of Debian Systems?

I’m sure there must be lots written about group system administration, but it doesn’t seem to be written in either the FAQ, the reference or even the venerable FDL’d SAG, so I hope it’s not a(nother) completely silly question. As I was reminded by the Paralysed Perl Package Problem, sometimes other system administrators can really mess you up by changing things without documenting what, how or why they made that change.

My current solution on that system is to put the message “please record any major system changes with the command dch -c /root/changelog --no-auto-nmu -i 'description of change'” in the /etc/motd file. I’ve also installed apt-listchanges with a suitable configuration. For TTLLP servers, there’s not a problem because we all use the same task tracker to make notes.

For shared/remote servers, I’d like to have something better than fault-finding and the intrusion detection tools, but stop short of trying to require all system administrators to use a particular version control on the system configuration, or trying to require them to use a centralised bug tracker application. (The other sysadmins work for other people, so we can’t require them to do it and “pay us to manage a repository/bug tracker for your server” is an awkward sell anyway.)

What do you do?

This entry was posted in GNU/Linux. Bookmark the permalink.

13 Responses to Tools for Group Administration of Debian Systems?

  1. -dsr- says:

    I’m not sure I understand your situation. You’re saying that you are looking for solutions for multiple root-privileged users who have no common superior, on a single box?

    I think you’re hosed. Divide the physical box into multiple virtual machines, let each root handle their own VM, and make sure there is one primary authority for root on the physical host.

    If there is a common authority supervising all these root users, then use that authority to enforce policy.

  2. Anton Piatek says:

    You make an interesting point – I manage several Debian boxes but as I am the only admin I don’t have any process for tracking changes but I really should start something so when it all goes wrong I know which box has had what done to it

  3. MJ Ray says:

    @dsr – yes, I’m looking for solutions for multiple root-privileged users where the common superior is a non-technical commissioner. I don’t think we can walk away from every time we encounter this and I think multiple virtual machines is as awkward a sell as change or task management. The common authority can’t enforce policy because there is currently no policy to enforce! Essentially, I’m looking for examples of good policy from elsewhere.

    @Anton Piatek – TruckNumber = 1. Be afraid.

  4. -dsr- says:

    If you can enforce a policy, then you need something along the lines of “all changes made by a root user MUST be done through Puppet” where Puppet takes on the value of your favorite systems management automation entity. Yes, this includes and is more authoritarian than your already discarded “force everyone to use the same version control system”.
    It is. I think, the only way to make this work.

  5. taggart says:

    We use a similar dch method, documented at

    http://lackof.org/taggart/hacking/dch/

    We’ve started using etckeeper as well, but only letting aptitude do autocommits.

  6. I essentially use the dch solution, but mine’s homebrew (a two liner shell script that echoes `date` and $LOGNAME >> /root/Changelog and runs $EDITOR on it). The other sysadmin(s) usually remember to describe their changes, although sometimes I have to recover those descriptions from vim’s .swp files.

    BTW it’s really useful to write down the exact shell commands/changes to config files you’ve made — you’ll find out that you need to do the same thing again (e.g. on a different box) and you’ll *love* the ready-made cheat sheet.

    I’m planning to look at etckeeper next.

  7. MJ Ray says:

    I’m inclined to start using etckeeper too. I’ve been hearing quite a bit of buzz about it over the last few days.

    About writing down the exact shell commands: do you think it’s worth starting up “script” or setting BASH_HISTORY to a one-per-use file, to save all root commands unless told otherwise?

  8. Markus Hochholdinger says:

    Not the perfect solution but an easy one. All sysadmins (root) have to stand to these rules:
    - If a configuration file will be changed the first time, do “cp -a $FILE $FILE.original”
    - If a configuration file will be changed if *.original exists, do “cp -a $FILE $FILE.$(date +%Y%m%d)”
    - Comment your changes in the configuration file, especially who changed it.
    => locate *.original, diff $FILE.original $FILE, ls -l $FILE, ..
    - Only install software with aptitude
    => /var/log/aptitude*

  9. jhr says:

    mj, you mentioned fault-finding tolls. do you know please something like lintian, but for system with the posibility to write simple policy checks?

    -jhr.

  10. MJ Ray says:

    jhr, I wish I did. systraq, diffmon or intrusion detectors will report changes, while slack, cfengine2 or puppet can be used to enforce policy, but I don’t know of a checker quite like lintian. Maybe lintian or linda could be repurposed without too much effort?

  11. I don’t use script or shell history — I copy and paste between two terminals, one running a shell, the other running vim. The things I record in /root/Changelog are idealized versions of the actual commands I run, and only those commands that actually make persistent changes (so, things like looking around with ls, reading man pages, fixing command lines arguments after a mistake—these aren’t recorded.)

    I also add human-readable comments in places that I think are warranted (e.g. vi /etc/cron.d/somefile, would be followed by “added a line” and the line itself: “0 * * * * * root /usr/local/sbin/somecommand”). Sometimes I use ex commands to indicate simple changes (vi /somefile; :w %.orig; :%s/foo/bar/gc). In other words, it’s more of a collection of recipes for sysadmins than a changelog — and it seems to work pretty well. I use it as a recipe more often (we need to to the same thing for customer X’s instance that we did for customer Y last year — let me look it up) than as a debugging/audit tool (but it is also useful in the latter case—e.g. some mail processing script stopped working last Tuesday, did anybody change anything at that date?).

    Sometimes I prepare the commands in vim (e.g. using copy/paste from older historical records) and then paste them into shell, one by one.

  12. reg says:

    In my sysadmin team, we use metche, apt-listchanges, apticron and a custom script which insert changelogs in a central database (script is automatically launched when exit).

  13. Matthew Edmondson says:

    I manage several different shared root Debian systems.

    I will look into the tools documented above. Thankyou.

    For each group we develop a group policy about ‘being root’ and how that impacts on documentation and changes to the system. Words like ‘Absolutely must’ are used. Before work is done, people generally propose the changes, prior to doing any work that may have impact on others. The ethos being responsible co-operation.

    Documentation is required and has been done in a variety of ways, inlcuding on an internal facing wikis on the box.

    Using sudo is commonly a policy.

    This method has enabled me to mentor new admins and thus devolve work in maintaining voluntary projects.

Leave a Reply

Your email address will not be published. Required fields are marked *




* Copy this password:

* Type or paste password here:

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>