Linux Czar

Been reading about the aspects of physical security. Would like to do some more reading on the subject. There are quite a few short papers out there that talk about aspects of physical security or give you some type of checklist aimed at helping you cover all the bases in your data center. One, entitled Data Center Physical Security Checklist, does the checklist thing quite well. Its very good about confining its scope to physical security and is quite professionally done.

This list has given me some good idea about things to do about my server room and, hopefully, more points to take to the faculty regarding how unacceptable the current condition is.

Just to top off facilities randomly chose yesterday to get the vendor out to do some work on our AC system. I have had a call in about some malfunctions we have been having recently and the work most definitely needed to be done. However, facilities did not give us any notice. “Hi, we are starting work on the AC system, the physics department told us we should talk to you.” Good grief! We are now back up after a two day degradation. I didn’t quite turn everything off, as the basic building AC will hold us to a point.

The physics folks move into the server room’s outer room next week.

I feel that this should be told in the form of an epic story. But I’m too stressed out right now to add such a flare. A day that I’ve seen coming for a very long time has finally come. Well, come and past. It was just today that I was able to put all the peaces in order and connect all the dots.

We have a server room in the basement of the building I work in. The space used to belong to physics. There is an inner room that is the server room and another room you must go through to get into the server room. From the outter room, the server room is not secure. The inner door can be opened with a coat hanger, there are vents between the two rooms, bad dry wall, etc. Its damn cold because of the AC in the server room, and loud. Also, the physics department still controls the keys to this room and gives far too many folks access to what should be a secure area. We also use the outer room for receiving shippments and as a staging area for upgrades of labs which happens two or three times a year. But, the main point is that if the outer room is physically compromised so is the server room.

The day has come where physics wants to move grad students into this outer room. We have made folks aware of other space in the building 10 feet away from this one, but that is too far away from the research group. I’ve had the bossman make the point of the aspects of physical security of the server room. No argument was able to sway the physics department. The deans and administration refused to back our claims. The bossman is now having our student helpers clean out the outer room and move stuff into the server room. This is creating air flow problems already. Did I mention a fire hazard? Its very bad practice to clutter a server room.

Yet, the battle has been lost. I have no idea when students will move into that space but that room as been re-aquired by the physics department.

What do I do with a physically compromised server room? What do I do when some physics minion opens the server room being cold and tries to adjust the thermostat? What do I tell the users of the equipment in that room?

Why will the administration not back the efforts of the IT professionals? I have no budget and am unable to aquire the expertise and equipment to do my job at what I would think would be satisfactory. No matter what I say, what I do, what policy I try to create the faculty always turn everything around so everything is the fault of the IT professionals. Even a storage/backup catastrophe from a completely different IT group is blamed on my IT group.

There are many things that piss me off. There are many things that make me think about getting a new job away from the university. There are a lot of things wrong that work hard to correct. I can deal with a lot of political mess. But the faculty have gone across the line. There is no turning back. This situation is completely inexcusable. But, I still have no way to convice PhDs that there is something wrong here.

I have an interview with Red Hat tomorrow.

Quota is always painful. But its not as hard a problem as, say, what StateEngine tries to solve. Always raising a user’s quota when they hit the invisible wall is not practical. In the Linux world, or at least over here at NCSU you get a dialog box that says you are over quota and brings you into Failsafe mode for your graphical login. Failsafe mode, yeah, that mode where you have a crappy xterm in the bottom of your screen. I have a lot of users that have no clue how to use a Bash prompt.

Why hasn’t anyone solved this user interaction problem? Why is there not a generic tool that can pop up letting users examine their files and delete some of them. Some limited file management aspects. I realize that there are many different file systems and different ways for querying about your quota. This can be pretty easily abstracted into modules.

This is one of the biggest non-automated tasks I am bothered by. This should be an easy thing for users to fix themselves.

Something was suggested to me and its true. There are two types of System Administrators at the University. There are those who understand the importance of the computer systems working together in unity. Where the entire system serves the needs of its users, is secure, and dependable. The second type of administrator is a person that believes they are hired to fix computer problems and make people happy.

In all cases, the individual computer as well as a complex IT infrastructure is meant to serve the people at the organization. However, there are many times when a user must be told “No” because what they request would degrade the performance of the IT infrastructure or degrade the services that IT is required to provide. In many cases the administrator must be aware enough that they need to understand the user’s problem and to be able to find a solution that does fit into the IT infrastructure. If you see your job as to make anyone and every happy then you will only provide the solutions dictated by the users rather than understanding and solving the actual problem.

When asked about a particular user, in the context of retrieving a file from the backups, a system administrator was not sure that the user’s computer was being backed up at all. He replied that one of the things on the list was a backup solution for the departmental staff. He also asked if we had any suggestions on what he could deploy “other than all going back to a distributed computing solution.” Yikes.

Unfortunately, it seems that most of the “system administrators” at the university are of the second type. Unable to use the existing systems that we have used to create and restore backups for years. Unable to think beyond what a user told them to do. Unable to grasp the larger concepts that keep IT working in a sane and manageable way.

Now I get to watch as the second type of computer administrator tears down a well build system that’s been proven and well used for years because a user said they didn’t want to have quota.