Current Version 1.7.2 has been released. Get the latest packages and source from
http://current.tigris.org
Current is a GPL’d implentation of an Up2date server for Red Hat style distributions. This developers’ release focuses on creating a model and classes to handle web page templating.
I found myself unwillingly in a discussion about security or lack thereof. The security of certain tools that I have written and rolled out to about 1,000 workstations and servers.
The details: I create and support a RHEL based operating system that’s installed throughout campus. Its right under 1,000 systems and is built to scale to many more. A Systems Administrator can install trusted, supported versions of the product as well as random folks in dorm rooms can install an untrusted, unsupported setup. (Although, the latter has declined quite a bit.) If you are a Systems Administrator you have write access to a directory in AFS where you can put a config file that controls dynamic generation of a Kickstart file for a specific system. Then you can install said system. There are, roughly, about 50 such persons. Our security officer approves these persons, not me.
So far so good. Systems Administrators have a pretty cool way to install a “supported” system in a completely automated way. The config files allow for a very highly flexible and configurable installation. Then comes the fun part. There is a default root password and administrative user list that must be applied to the system. Each group on campus can also have their own root password and user list that overrides the defaults. These, mostly the root MD5 hash, are considered secret and are not shared. Being that MD5 hashes are easily brute forcible, they are encrypted. For each of my machines, if its s trusted machine it must have the write root password and users list including needed updates. If its an untrusted machine it must not have access to the secret information. Finally, the installer may have root access on the system, but that does not mean he or she knows the root password.
This problem boils down to distributing a group name (like ‘foolab’) and an encryption key. Once you have this information the encrypted blobs are world readable in AFS and are easily applied to the system with a little sysadmin know how.
My system works with a registration database and public/private RSA keys on the machines. Once installed a machine creates its keys and registers with the database. Then, that machines public key, signed by its private, serves as a positive identity that cannot be faked.
But, that’s not the problem. The problem is when machines register. Even though there are a specific set of circumstances that must be met before a machine will be allowed to register, it is still a completely automated process. In that window before a machine is successfully registered I do not know if that machine is the machine setup and installed by the Systems Administrator or if it has been replaced or spoofed with an imposter.
The system is completely automated. It cannot be secure. Push connections from the database or pull connections to the database. It cannot be done. There is obscurity involved. It bothers me. Its also based on the same systems used for other UN*X systems on campus for years. Its flawed. Asking for someone to have a little fun.
As a friend of mine states, and you’ll not find a better definition of security anywhere, security is based on one of three things. 1) You know something. (A password.) 2) You have something. (A key for your car.) 3) You are something. (A Systems Administrator.) Before machine registration none of these are true. You may generate all the RSA keys you want, the above must be true for the database and the client.
In such an automated process the only thing you can do to make the system end to end secure is to have one of those 50 or so Systems Administrators come by and bless the machine. (Also known as make_me_not_a_virgin.py.) This operation makes the system secure because the System Administrator (You are something) takes the public key for the machine and puts it in an AFS ACL’d place where only Systems Administrators can write and is loaded into the database. Then you have something. The database can securely recognize the machine and feed the machine the needed secrets in an encrypted way that only the machine that was blessed can decrypt.
The trick is to make that script you run to bless a machine as slick as possible. The goal being to make it easy. The downside is, how to convince folks that machine blessing is good. Having to touch each machine is quite frowned upon.
Well, that export option helps. But it really doesn’t solve the problem. Check out
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=158293
After much searching and continuation of my NFS problems I think I might have found a workaround. This looks to be a evil bug in the 2.6 Linux kernel. I was able to find information in a LMKL thread from November. However, I haven’t seen confirmation of an actual fix. Don’t know if its been fixed or not. In looking at the RHEL 4 kernel errata that are available for my production NFS server I don’t see any changelog information that looks to address the problem.
That LKML thread did suggest using the no_subtree_check option in your /etc/exports file. Indeed, that does appear to hold off the problem.
Well…maybe…if I could only cold start the damn system.
Today at work, instead of doing something useful, I’ve been beating my head against NFS in my Beowulf. I have 3 or 4 very special users that get their home directory mounted by autofs. On the very next cycle when autofs goes to try to unmount unused home directories these specific mount points go stale. The ls command reports no file or directory. I can cd into these home directories and all I get is a stale NFS handle. My log file has the most interesting part:
nfs_statfs64: statfs error = 2
Google turned up “no matches.” I guess it wont after it indexes this post. But I was completely baffled by that. I’ve no real clue to what this error indicates. The obvious part is that my RHEL 4 NFS server doesn’t get along with RHEL 3 and autofs.
I know NFS is tricky business. But normally, it interoperates with itself pretty well. The only thing I can do here is completely disable the automounter and just use a big NFS mount for /home. I really hate not knowing why.
At work we have rolled out RHN Satellite. Well, I have rolled it out. With some Python and RPM foo I was able to cleanly migrate 500 or so clients over to the Satellite. Only had to clean up 30 or so. Not too bad.
I admit that this whole thing has really been frustrating. There is no real migration path to take a few hundred machines in 30 some odd groups from rhn.redhat.com and migrate them to your own Satellite. That was a lot of work. Some good code envolved.
Now that we are over to Satellite all the Red Hat quality crap uncovers itself. Passwords stored in a local log file. I’ve got the Satellite configured to use PAM to integrate into our Kerberos system. Passwords with certain characters just don’t work to register machines. Up2date just kinda hangs. Takes an Apache process on the server and pegs it. Basically, sometimes you log in, sometimes you end up pegging server processes, and, if it doesn’t like you, it wont let you register a machine.
Oh yeah, and then there was the 10 minutes when it sent out like 1,000 emails. The anti-spam systems blocked it from the mail relays. It was sending out errata notifications for RHEL 3 Update 5. At least 30 emails per user account. Any sort of mode to throttle that traffic and basically be a polite internet citizen? No.
Add in keeping up with Oracle 9iR2 for the back end database. Which is actually giving me the least amount of worry. Backups too.
Is all this worth it for the nice RHN interface? I’m definately going to say no. When you realize what a shabby product RHEL has turned into, and all this mess to manage it, its just not worth it. I’m so tired of beating my head against the horrible, show stopping bugs that are everywhere.
Want a real solution? Use CentOS and Yum. Its cheaper. Its better quality and more stable. (Yes, I know its RHEL rebuilt…that’s actually how it gets QA’d better.) It Just Works(tm).
Really…to reiterate. The future of package management systems like RPM and dpkg is a system like Conary. The company maintaining Conary has recently under gone a name change to Rpath.
Also, in the ramblings of my own mind…I have to wonder about the possibilities of combining Conary with the would be features of Current. An RHN-alike system for a freshly born system management tool. Hmmm…
Where would we be without science fiction? Probably in the dark ages. So much of our day to day technology that we take for granted was originally thought of in some form of science fiction like Star Trek. Do you talk into your flip phone like Captain Kirk? I thought so.
Unfortunately, I’ve yet to read a good book with an imaginative and futuristic approach to package management. I just don’t understand why this couldn’t be on the best seller list. Well, maybe I could. So, what’s with RPM, dpkg, tar packages, portage? Its all ancient technology. I mean, this is so 80’s man!
They are all slow. Have you installed an RPM lately. Wow it takes for ever.
We’ve been patching source code with the ‘patch’ command for how many years? Yet when a new OpenOffice.org security fix comes out we get to down load how many megabytes to fix a one-liner?
Conflicts, Obsoletes, Requires…oh my! RPM has gotten amazingly complex over the years. Every time there’s been a new packaging problem RPM has had a whole slew of functionality patched in. Have you ever dealt with multilib packages with RPM? Quick! Somebody claw my eyes out! Now we have versioned obsoletes, implied obsoletes. When folks joke about running “rpm –emacs” I really am surprised that doesn’t work yet. If you’ve had a good run at package management and a good tool like Yum or Up2date you know that solving dependencies is no simple task. That’s hard code.
One of my very favorite problems with packages is what happens if you want to make a change or need to for some reason. You have to roll a new package with your changes and push it out to your clients. You have to deal with what happens when the Red Hat Network pushes out a higher version that your package which effectively undoes your changes. In fact, you have to maintain your very own, completely independent branch for this package. Now what if its one of those Fedora Core packages from hell? Try the perl package, or sendmail.
Fortunately, there’s light at the end of the tunnel. Well, maybe somebody with a match and a magnifying glass. Put on your graph theory cap. Think, from a high level, what are we doing with package management? Hint: we do with all the time with our source code. Answer: Change Management. Definitely a different problem subset than what CVS, Subversion, or any other SCM I know of, but it is in the same superset. Think of a tree or graph of package objects. The graph contains each file. Like any SCM you can retrieve any version of a file or package object. Now think about having a set of these package objects installed on your machine. You can control what versions you have installed. A local change in an object becomes a node in your local graph. Think about updating an object and your machine dealing with a change set of exactly the one line that changed upstream.
Makes your brain hurt doesn’t it? Is it worth something? You bet. Is it complex? Yeah, that too. Is it the future of package management? I honestly don’t know. But its designed from the ground up to deal with today’s package management problems in a sane way. Its designed to be extensible into the future. Its designed to help folks make changes and modifications to Linux distributions and not be forced to completely branch off from your vendor. Its called Conary. You should learn about it.