Linux Czar

I was smiling as my new Dell 1950 server installed. A replacement web server, its configuration is more complex than most things. It installed beautifully via the automated installation process. How system administration should work. Then came its first boot. It printed ” Bad PBR signature” and stopped.

Looks like this appears to be something with the MBR on the drives that I’m attempting to setup in a RAID 1. So, let’s re-install Grub per the suggestions I got off of IRC. I did. I keep a Grub boot CD handy. No matter how I installed grub the machine wouldn’t boot. I never saw the above message again, but instead received clutter, random bits of what must have been the Dell utility partition, or “Grub loading stage 1.5″ over and over and over.

Alas…nothing can ever be easy and simple. I could have gotten something done.

Hmmm…you know, I’ve had complaints from folks installing RHEL 4.4 on a RAID 1 system and having Grub problems. I chalked it up to BIOS issues or not understanding what’s going on. But, perhaps there is more to Bug 217176.

I run ftp.linux.ncsu.edu which is a FTP/HTTP mirror of Fedora, CentOS, and other things I find useful. Apparently, my HTTP side has been “discovered.” The pages day in and day out telling me my server was down quickly became annoying. Especially as this was not the server being down put being DoS’d by your average download accelerator. Like the FTP side I wanted to limit the mount of connections each IP could have so the next time I got out grep I wouldn’t find on IP connected 63 times to download an ISO.

Apache itself doesn’t do this without 3rd party modules. Looking more closely at my RHEL 3 server I find that the IPTables tool has the ‘iplimit’ module but the kernel isn’t built with that module. The kernel is built with the ‘recent’ NetFilter module, however the IPTables tool is not. Good grief, Red Hat. (I also noted that RHEL 4’s IPTables have the ‘connlimit’ module but that kernel does not have the matching module either.)

I decided to use the NetFilter tools and grabbed the IPTables source RPM from RHEL 4, rebuilt, and installed that on my server. Now, I can use the ‘recent’ match. I typed in the following rules:

1. iptables -A INPUT -p tcp –dport 80 –tcp-flags FIN FIN -m recent –name httpusers –remove -j ACCEPT
2. iptables -A INPUT -p tcp –dport 80 –syn -m recent –name httpusers –rcheck –seconds 120 –hitcount 5 -j REJECT
3. iptables -A INPUT -p tcp –dport 80 –syn -m recent –name httpusers –set -j ACCEPT

These rules are very order specific. The first rule checks to see if a HTTP connection is closing and removes the IP from the list. The second rule checks to see if this is a new connection to the http port and looks to see if there are 5 other recorded SYN packets from this IP in the last 120 seconds. If this matches (more than 5 connections) the packet is rejected and the user informed with an ICMP packet. Finally, all new connections are recorded in the IP list.

Is this perfect? Far from it. I’d really like to have the ‘conntrack’ module built for the kernel to do this right. If one connection closes in the above example the connection counter is reset to 0. However, this should level the playing field against those script kiddies.

Its been in the rumor mill for some time. However, today was the first time I’ve seen it reported by a news outlet and I definitely think that its worthy to repost here. Red Hat has plans to release the RHN code as Open Source.

• Red Hat to open source its Network product - InfoWorld

Something I am very excited about. Even as the maintainer of Current. The rumor mill has it that this will be RHN 5.0.0.

Things I’d really like to see in an open source RHN are support for open source databases such as PostgreSQL or MySQL. Oracle 9iR2 is a little dated. A conversion path is also doubleplusgood.

I wrote up a short wiki document about setting up and using the device mapper multipathing tools and LVM in Linux. Its targeted at my work environment and our EMC Clariion. However, I thought it might also be useful for others. Especially, if you suffer from the horrid PowerPath packages. Must suppress the need to rant. Also, includes instructions on the proper device nodes to use in your fstab which solves the “timing problem” that some folks have reported.

• http://www.linux.ncsu.edu/moin/RealmLinuxServers/Multipathing

Email is a great way to stay informed about what’s going on in the Linux world. I get probably around 500 - 800 emails a day and, unless I’ve stopped reading the list, at least know that something has passed across my radar. Enough so that if a question or situation comes up I remember that I’ve gotten email about that.

Email is a great way to get pissed off too. I just received a bit of bugzilla email for Bug #150828. For the life of RHEL 4 (close to 2 years now) PHP and perl/python modules have been compiled against different versions of the MySQL libraries. These are very important parts to running any sort of web application. There’s even an acronym for it, LAMP. So if you load two incompatible MySQL libraries into Apache (say you run some PHP code here and some Python there) guess what happens? Apache segfaults.

This isn’t an Apache problem, this is various bits of RHEL 4 being built against different versions of libraries. The fix is simple. The python-MySQL package is built against the lesser version of MySQL and I rebuilt it…2 years ago. Yet, to this day Red Hat has not been able to correct the bug. Its always promised for Update N+1. No reason for the delay is given.

Frustration. Once again I have to carry forward customizations that should be unnecessary.

Finally. Its started…

• The New RPM

The work place is sometimes a really interesting place to be. We come from a strong Solaris background and in the last 6 years or so have migrated many of our mission critical services to Red Hat Enterprise Linux. However, for the foreseeable future, we will be running a mix of Solaris and Linux.

Solaris sucks in that it doesn’t come with much. Apparently, when Sun ships you a broken compiler that is considered a leg up. The natural thing to do is build tarballs of applications to distribute or just the old fashioned untar, configure, make, pray, make install.

Linux distributions, on the other hand, are very different. They normally come with a “batteries included” attitude toward common software. So you’ve got plenty of perl, berkley db, MySQL, python, etc. It naturally leads to building packages with, in our case, RPM.

So with many IT employees each building the software they need for Solaris and augmenting Linux as required it didn’t take long for our servers to become very different and incompatible with each other. As we continue to be a mixed shop we constantly deal with upgrading services from Solaris to Linux and, most commonly, interactions between Solaris servers and Linux servers to deploy a service. With no method to our madness it became virtually impossible to interroperate with servers (Linux or Solaris) that other people built and run.

Being that we have to maintain our compatibility between Linux and Solaris (especially for things like Perl and Berkeley DB) how do you create unity in our environment? Very frankly, I’d really like to know how other folks have tackled this or similar situations. What ways work best for you? What problems do you encounter. What problems are you specifically trying to solve?

At NCSU the solution that folks came up with quite a while ago is called “The Vision.” It has successfully created a standard way of building applications that works for both Linux and Solaris and has solved compatibility problems between the two. However, its not really the Linux Way(tm) and far from the Solaris Way. In fact, I really have not seen such implemented like this elsewhere.

Software is built with RPM for both Solaris and Linux. It all installs under /local which definitely makes me twitch. Being specific to NCSU (the /local is good enough reason to do this in my book) packages are named ncsu-packagename. There are a set of guidelines for writing spec files and make files which do the building. (They enforce the RPM macros are always the same.)

Oh dear, does it make me twitch. But its also jaw droppingly amazing. For our applications and scripts, and most definitely as many things are powered by Perl, this has created a sanity and peace between Linux and Solaris. Perl scripts run everywhere instead of having to deal with Perl module hell and why is the modules installed in place X over here and place Y over there. Thie Vision has created a stable environment and allowed folks to transition to Linux easier and faster and have much more sanity in dealing with administrative tasks on all operating systems. In fact, (I’m not kidding) there is one rpmroot that contains the sources, specs, and packages for both Solaris and Linux. Each spec file builds on both RHEL 4 and Solaris 2.8.

As the Linux Czar for campus I’ve always done things the Linux Way and pushed the Linux agenda. I know folks will read the above and think that we must be the worlds top crack producing IT organization. My servers that provide Linux related services to campus are built the Linux Way rather than the above. Its also probably clear that The Vision is not new but that I am truly new to it.

I find myself in a position where I can participate in steering The Vision. I honestly have never been so horrified and in awe of the work that has been done before me and the responsibilities I now share. The Vision has brought about something amazing to our IT organisation. Indeed, there is forward progress to be made and new things to accomplish. So, this begs the question, where and how do I help steer The Vision?

How would YOU have solved this?

Where would YOU go from here?

In trying to drum up a little publicity and community involvement for Current I posted this on the Current Announcements page which I hope will appear in Tigris’s main announcement stream.

On October 25th Oracle announced Unbreakable Linux and the Unbreakable Linux Network (ULN). The ULN is Oracle’s web based application to manage licenses and software updates for Unbreakable Linux. Under the hood of the Oracle Apex web interface to the ULN lies a small open source project called Current.

Current is a tool to deploy and manage the sets of packages installed on an RPM based Linux distribution. It has been used in production environments with thousands of Linux clients. It works with with Up2date and newer versions add support for using Yum against a Current server.

Originally, Current was designed to be an open source Red Hat Network (RHN) server. Today, the Current developers believe in creating a much more powerful and useful tool. We believe that there is value in strongly authenticated package repositories, an advanced web management interface, client tracking tools, and a flexible system for grouping and permissions. Many of the exiting management tools limit our freedom as they are controlled by corporations and closed source. Other open source tools lack strong authentication and management features. Simply, we believe that the Open Source community can create a significantly more useful management and deployment tool.

Current is licensed under the GNU General Public License and written in Python. As of this writing the latest stable version is 1.6.1 and the development version is 1.7.6. While several entities have created web interfaces for Current, the Current project’s web interface is in the very beginning stages of development. The Current project is actively looking for contributors to help develop the web interface. For more information please see the project’s web site at http://current.tigris.org

The apocalypse. I’ve always known I’ll have a hand in it. — Hunter

Too true, Hunter. However, now that I’m the current maintainer of Current its all my fault. I’m sure if you are bored enough to read this than you have heard of Oracle announcement of Unbreakable Linux. Included with that is Oracle’s Unbreakable Linux Network which is the RHN-alike.

If you know where to look (besides here, obviously) its no secret. Oracle’s Unbreakable Linux Network is based on Current. The web user interface was created with Oracle’s Apex product. Fortunately, I seem to have a fairly decent relationship with a few folks in Oracle’s linux team. They’ve been hacking on Current and asking questions for a while. I’ve even been sent patches with the promise of more to come. They have been working on Current in good faith and I really appreciate that.

Most of the changes Oracle has made are in porting Current to run with an Oracle database. Unfortunately, with the way the database layer is designed and the differences that Oracle requires I can either have support for an Oracle database or have support for SQLite, MySQL, and Postgres. So those patches wont appear in Current’s SVN tree anytime soon. However, I’m definitely still interested in solving this problem the Right Way(tm) by porting Current to use SQLAlchemy. I’ve definitely told folks I will accept such a patch. However, there’s enough SQL in Current to make that quite a bit of work.

For the truly bored with way to much free time, there is one other thing that I’d like Current to have. A suite of unit tests aimed at the external XMLRPC API and the directory structure that Apache serves out via Current.

I definitely think that there are interesting times ahead for Red Hat and the Oracle Linux offering. Its obvious that Oracle has no desire to put Red Hat out of business as their linux offering is completely dependant on the continued maintenance and new releases of Red Hat Enterprise Linux. If you scratch off the Unbreakable Linux sticker you’ll see RHEL underneath. We all know that there have been rumors for months about Oracle wishing to acquire Red Hat.

I believe that this could really put Red Hat back in the game on the competitive edge. Perhaps Red Hat needs a good bit of competition. Working at a university I know well that Red Hat’s support offerings are very expensive and they are sold on the subscription model. Red Hat, the subscription model by itself just isn’t competitive and doesn’t allow my place of work to purchase support at a reasonable cost level. How about offering different support schemes and let your customers buy what’s right for them. Sell support instances. Let me buy a 10-pack. What about support contacts? Pay a flat rate per year for one person at an institution/company to open as many support instances as needed. Make a bigger deal of your web based support products. Quit porting RHN to Java and make it the management tool of the future.

The last couple weeks have involved a lot of me banging my head on my desk trying to make printing sane. We have used LPRng at NCSU for quite some time now. It scales well and it supports strong authentication via kerberos. Authentication is important as users buy print quota with a credit card, normally 6 cents per black and white page.

However, LPRng has become rather unmaintained and Linux and other platforms are moving in the direction of Cups and IPP based printing. Even more, Cups provides a printing API that GUI applications, like OpenOffice.org, are starting to use. So I am quickly approaching a divide where CLI based printing still works but the GUI applications insist on printing to Cups which knows nothing about my kerberized LPRng printers.

Cups is currently not deployable mainly due to the lack of strong authentication. It supports PAM authentication which is an attempt in the right direction. However, who wants to enter their username and password for every print job…that’s when the local print drivers know what to do with Apache style basic auth. That’s more broken than functional.

I’m getting into more and more situations where if somebody has set up a Cups browsable printer near a lab that breaks printing for the entire lab. So what to do as Cups develops kerberos support and better scalability? We write our own driver. Halloween coding style. Its some scary code.

• ncsucups.py

In a nutshell, it runs as root and nukes things that Cups puts in LPRng’s way. Then it changes to the user that ran the print job, finds their authentication bits, and runs lpr from our local LPRng distribution. Some impressively bad code, but it is successful at integrating Cups onto our printing system.

While I’m working on all this I’ve also been poking around Cups as I know well of the Google Summer of Code project to add proper kerberos support for Cups 1.3. Oddly enough, today I was made aware that apparently the work is done. Today I was made aware of the rumors saying that Cups 1.3 will be released very soon. *sigh*

To top things off looks like there’s some LPRng code that supports kerberized IPP too. Its really bothersome how powerful LPRng can be yet I’m almost being forced to move away from it. Another thing I’m interested in is pulling printer information out of LDAP and into Cups. However, Cups doesn’t seem to understand that LDAP is a read only database, it seems to want to use LDAP as its own data store.

In any case, I look forward to testing Cups 1.3 and seeing what’s new and what’s different.