My 100-line Python program

matthew's picture

I've been using Python for a few years, but mostly for extremely simple and short stuff. By and large, truly, I've mainly just referenced someone else's libraries in Python to do something that I couldn't do easily in Bash. Today, I decided to finally knuckle down and undertake a conversion script I normally would have done in Bash using grep, awk, and sed, and instead do the whole thing in Python.

Now, I realize this isn't very Pythonic. It's quite procedural, no classes or functions. There are several sections where I think I could dramatically shorten it by splitting some of the more repetitive areas out into classes. But for a first real attempt at using Python as a replacement for Bash shell scripting, it was fun!

Takes input from a file with a format like this:

filername-nfs:/vol/some_volume_name/some_qtree_name 20G   private on some_hostname  mounted at /some/path/to/nfs/mount

Now, the goal here is to automagically spit out a series of commands that I can copy/paste and run in various windows. I need it to calculate the size of all the qtrees in a volume and add 20% (divide by 0.8) and give me an integer to use in gigabytes. So imagine you just received a file containing, say, fifty different mounts in this format for perhaps 10 different hosts and two or three different Netapp Filers. At that point, the time savings become obvious! Using just the one sample above, the output from this file looks like this:

Volume size commands:
rsh filername vol size /vol/some_volume_name +25G

some_hostname /etc/fstab entries:
filername-nfs:/vol/some_volume_name/some_qtree_name /some/path/to/nfs/mount nfs rw,bg,nointr,hard,timeo=600,wsize=32768,rsize=32768,nfsvers=3,tcp 0 0
some_hostname mkdir & mount commands:
mkdir -p /some/path/to/nfs/mount
mount /some/path/to/nfs/mount

filername /etc/exports entries:
/vol/some_volume_name/some_qtree_name -rw=some_hostname-nfs:root=some_hostname-nfs

filername /etc/quotas entries:
/vol/some_volume_name/some_qtree_name tree 20G

Here's the script.

convert.py #!/usr/bin/env python
# Short little script to awk up things and do replacements in several files.
# Basically the same thing I do from bash, but in python.
import sys

# Debug this session?
debug = False

# Set the options we'll spit out for mounts.
# Typically, you don't need to alter this.
# But for Sun 7000-series filers, you need to add ",noacl" after "tcp"
nfsOptions = " nfs \
rw,bg,nointr,hard,timeo=600,wsize=32768,rsize=32768,nfsvers=3,tcp 0 0"

# Name of the file we're working from.
myFile= open( "qtrees.txt", "rU" )


allQtrees=[]
for myRow in myFile:
    try:
        # Filter out empty lines
        if not myRow.isspace():
            # Filter out comments
            if "#" not in myRow:
                # Remove newlines
                rsplitRow=myRow.rstrip('\n')
                # Split into space or tab-delimited fields like awk.
                allQtrees.append(rsplitRow.split())
    except ValueError:
        pass

# Initialize a few variables...
mountContainer = {}
mountFiler = {}
mountVolume = {}
numRows = len(allQtrees)
for column in range(numRows):
    # 0: nfs path, 1:size, 2:private/public, 3:, 4:host, 5:, 6:, 7:mountpoint
    # print allQtrees[column]
    myColumn = ( allQtrees[column] )
    fullNfsPath = myColumn[0].split(":")
    nfsFiler = fullNfsPath[0]
    nfsPath = fullNfsPath[1]
    nfsVolumeAndQtree = fullNfsPath[1].split("/")
    #print nfsVolumeAndQtree
    nfsVolume = nfsVolumeAndQtree[2]
    nfsSize = myColumn[1]
    privateOrPublic = myColumn[2]
    nfsHost = myColumn[4]
    mountPoint = myColumn[7]
# We're creating several Lists of Lists to be put into Dictionaries here.
# Main purpose is to make it easier for me to visualize the data, I'm sure
# there's a more efficient way than creating all these structures.
    # Wrap this list by hostname
    try:
        listByHost = mountContainer[nfsHost]
    except:
        listByHost = []
    listByHost.append([nfsFiler, \
        nfsPath, \
        nfsSize, \
        privateOrPublic, \
        nfsHost, \
        mountPoint])
    mountContainer[nfsHost] = listByHost

    # Wrap this list by filer
    try:
        listByFiler = mountFiler[nfsFiler]
    except:
        listByFiler = []
    listByFiler.append([nfsFiler, \
        nfsPath, \
        nfsSize, \
        privateOrPublic, \
        nfsHost, \
        mountPoint])
    mountFiler[nfsFiler] = listByFiler

    # Wrap this list by volume
    try:
        listByVolume = mountVolume[nfsVolume]
    except:
        listByVolume = []
    listByVolume.append([nfsFiler, \
        nfsPath, \
        nfsSize, \
        privateOrPublic, \
        nfsHost, \
        mountPoint])
    mountVolume[nfsVolume] = listByVolume

# Unwrap mountVolume for a list of volume sizes.
for volume in sorted(mountVolume):
    volSize = 0
    print
    print "Volume size commands:"
    for size in sorted(mountVolume[volume]):
        qtreeSize = int(size[2].strip('G'))
        curSize = str(qtreeSize)
        #print "Qtree size of " + size[1]  + " is " + curSize
        volSize = volSize + qtreeSize
        #print volSize
    volSize = int(volSize / 0.8)
    volSizeStr = str(volSize)
    print "rsh " + size[0].strip("-nfs") + " vol size /vol/" + volume + " +" \
            + volSizeStr + "G"

# Unwrap mountContainer for a list of mounts by host.
for host in sorted(mountContainer):
    mkdirCmds = []
    mountCmds = []
    print
    print "%s /etc/fstab entries:" % (host)
    for mount in sorted(mountContainer[host]):
        print mount[0] + ":" + mount[1] + " " + mount[5] + nfsOptions
        mkdirCmds.append(mount[5])
        mountCmds.append(mount[5])
    print "%s mkdir & mount commands:" %(host)
    for path in mkdirCmds:
        print "mkdir -p " + path
    for path in mountCmds:
        print "mount " + path



# Unwrap again for /etc/exports & /etc/quotas on filer
for filer in sorted(mountFiler):
    print
    print "%s /etc/exports entries:" % (filer.strip("-nfs"))
    for volQtree in sorted(mountFiler[filer]):
        print volQtree[1] + " -rw=" + volQtree[4] + "-nfs:root=" + volQtree[4] + \
        "-nfs"
    print
    print "%s /etc/quotas entries:" % (filer.strip("-nfs"))
    for volQtree in sorted(mountFiler[filer]):
        print volQtree[1] + " tree " + volQtree[2]

if debug:
    for name in dir():
        myvalue = eval(name)
        print name, "is", type(name), "and is equal to ", myvalue

myFile.close()