My experience in system administration based mostly on Linux systems and started in 1993 in Joint Institute for Nuclear Reasearch (JINR), Dubna, Russia. My first project was a setup of small Linux cluster for HEP group. As time progress, I've been carry out many system administration duties. Among them:

• Linux clusters setup and maintenance
• DB administration (primarily MySQL)
• kernel maintenance for Clue-d0 cluster
• software deployment and cross-platform ports (OSF-Solaris-Linux)
• system installation, troubleshooting, configuration,

The most interesting project was based at FNAL where we setup a large Linux cluster based on user desktops.

Cluster linux environment for D0 experiment (Clue-d0) was born as an idea to use idle CPU(s) of user desktops for analysis purpose. The HEP experiments is always a collaboration of different Institution, Universities and labs who brings their hardware to fulfill a goal of the project. That leads to heterogeneous hardware configuration of the custer. Based on our experience we recommended some hardware vendors for our collaborators. The underlying OS was always RedHat-based distributions (including different version of RedHat and Fedora Core). The user accounts were managed throw LDAP server where each group had own LDAP slave we kept in sync with a master. To accomodate various hardware as well as D0 requirements e.g. XFS as underlying file system for RAID file servers, we customized the underlying kernel for our needs. All file servers were equipped with Gigabit NIC. The underlying OS patches were provided in a form of RPMs and distributed in a cluster through autorpm mechanism. All users were able to run their tasks in interactive and batch queues. By the time I left the cluster consists of 250 nodes and was proven to be reliable alternative to physics analysis farm.