SouthEastLinuxFest 2012 After Action Report

| No Comments | No TrackBacks

This last weekend I took the 3.75hr trip up to Charlotte, NC for the southeastlinuxfest. This was my first trip to SELF and I was very impressed by the number of good talks and presentations.


On Friday, I arrived a little after the conference got underway and spent the morning in the “hallway track”, talking with and meeting a good number of people.

After a rather hasty lunch with herlo. We hurried back to the conference where I helped him give his presentation on the GoOSeProject. That presentation went VERY well, we ended up passing out all 10 of the GoOSe cds we had.

Sometime on Friday I also went to a presentation on OpenShift. However, I forgot to write down the name of the presenter so I don’t know who gave it. My schedule says it was likely Russel Bryant but I’m not sure.

I then skipped out on the afternoon sessions to go to a business lunch with some associates. The evening’s after-party, however, was lots of fun. I got to know a lot of folks: zonker, SUEHLE, Zach, Brian (the two aussies, didn’t get their addresses/emails), and others.. A lot of fun was had, good music, cards and discussion.



The real meat of the conference for me came on Saturday. After the keynote which I arrived late to. I went to the “Demoggification” talk by John Rose. This talk was awesome. It was about a lot more than just removing the “useless use of cat”. I got the following tips out of it:

  • Use Here STRINGS

    Herestrings are like Heredocs but lighter. Instead of doing:

    echo "foo" | some_command

    try this:

    some_command <<< "foo"
  • The /dev/tcp and /dev/udp files in Bash.

    I’ve used these before, they’re a great lightweight way of testing connectivity. I didn’t get a demo written down so I’ll research and do a blog post on them later.

  • Brace expansion tricks

    The common use is just to spit out generated strings.

    mkdir -p foo/{bar,baz,qux}

    However, brace expansion can do much more (like replace seq)

    echo file{01..10}

    It can also do both at once

    echo {foo,bar,baz}{01..10} # < no space between
  • Regular expression matching =~

    Try this instead of a case statement if you only have one fork of your condition.

    if [[ $foo =~ (FOO|BAR|BAZ} ]]
      echo "found the string"
      echo "nothing there"
  • You can also do pattern matching on your variables.

    $ FILENAME=/usr/share/doc/tmux-1.6/examples/screen-keys.conf
    $ echo ${FILENAME%/*}
    $ echo ${FILENAME##*/}

I had seen a few of these tricks earlier when I attended a talk by Michael Potter called Seatbelts and Airbags for BASH.

How to Quit Your Job (or How to Fire Your Boss)

My favorite presentation was the Saturday talk by Ryan C Gordon on how to quit your job. As I was recently let go from my full-time position and have been working for three months as a contractor/consultant this one hit home a lot of the points I’ve heard elsewhere. It also did a lot for giving me the “PEP TALK” I have needed in recent weeks. I kept three whole pages of notes. Here is what he talked about in brain/notebook dump format:

  • Pluses to being your own boss

    • more money (potential)

    You are not tied to just a single line of revenue, your potential for growth/income is no longer limited by one budget.

    You’re not going to be stuck on a single project for long. There is more diversity.

  • Minuses to being your own boss

    • less money

    you are going to start out making less money, so jumping into consulting without a safety net (something I was lucky enough to have) is almost always painful.

    • no medical insurance, no 401k, no safety net

    • more Lumberghs

    You no longer have to deal with one Lumbergh, but many Lumberghs. Luckily you don’t have to deal with them for the long term.

    • more hustle

    Being your own boss is a LOT of work. If you don’t have hustle you’ll starve.

  • Find a vertical market

    If you find your niche in a vertical market, you then need to stand out from the crowd. Which isn’t too hard, you only need to “Be better than the dumbasses.”.

    Ideally you would want a market with a lot of money and not many people in it.

  • “Your product is named ‘You’”

    You are your product. Sell yourself. Work hard at selling yourself. Don’t sell yourself short.

  • Milk your contacts

    Finding work is the hardest part. Milk your contacts for as much as you can.

  • Do Open Source

    A github page (with good content) is an excellent resume.

  • Always build your resume

    Your resume should always be easily found. Mine is

  • Always be interviewing

    This helps your people skills. It also helps you sell yourself properly.

  • Your online persona should be awesome

    Don’t post pictures of yourself puking in your twitter/facebook account. Always be aware of what your online persona looks like.

  • Hustle, do, ship

    It’s of tantamount importance that you actually SHIP aka deliver a product. If you never deliver you will never be invited back as a consultant.

    I always like the way Dave Ramsey puts it

    Get up.
    Leave the cave.
    Kill something.
    Drag it home.
  • Always be learning

    Ryan talked about how he was playing around with Lua once and blogged about it. That blog post led to a contact being made because of his “Lua knowledge”. If you play with stuff, learn stuff and blog about it you will have a working resume.

  • Keep records

    Not only of your timesheets for customers, but of what you’ve worked on. Again back to the living resume.

  • Married == uphill struggle

    Your relationships will have a harder time when you are contracting as the stress can cause issues.

    It is paramount to find a good balance

  • Budget Carefully

    Not just your money but your time.

  • Ask for more than the job is worth

    Always negotiate. Always go up. Double your rate.

    No one will walk away from a number without saying something like “hey that’s too much”.

    Have a minimum price, if the customer can’t match it, walk away.

  • Schedule well

    Over estimate times two.


  • Fire customers if they suck

    “The customer is always right” isn’t always right. Fire the hard ones.

    I first heard this advice from the guy who wrote “The no asshole rule”.

    Have a contract breaking clause.

  • Never say you are running out of money

    Always present yourself to customers as if you are on good footing

    Ryan mentioned the talk F#@k you, Pay me!. Good points, but LANGUAGE WARNING

  • Hire a lawyer

    You should talk to a lawyer at least once, and have a relationship with one for when you get sued.

  • Hire an accountant

    Not only for tax purposes, but for investing your money when you start to do well. Remember you don’t have a pension or 401k anymore.

  • Business type

    Sole Proprietorship, Default and easiest. Most risk.

    LLC, little harder, more protection. Hard to sell (the business).

    C or S Corporation, most difficult, lots of tax considerations.

  • Self Esteem

    Work on your self esteem, you need it to sell yourself.

  • Contracts

    If it’s not in writing, it didn’t happen. No verbal contracts.

    At a minimum, follow up with an email to any phone conversation.

  • Get health insurance

    Without insurance, one trip to the ER can bankrupt you.

  • Take care of yourself

    Buy a nice chair, nice monitor and a desk which sits at the right height.

  • Don’t rent space

    If you don’t have a space in your home, don’t rent/lease space. Find a coffee shop. Many multi-million dollar companies have been founded/sold without ever buying or leasing office space (more so in recent years).

  • Stick it out

    Don’t give up. Working for yourself is hard work and takes time to build a base. Hard work most often will turn around down times. Get out and pound the pavement.

    However, you should have an escape plan/failsafe. Start with a ‘failure condition’ in mind.

  • Have balance

    You need to balance your work, play, sleep and love. When any of these don’t get the attention they deserve you’ll end up burnt out rather quickly.

This morning I got bit by what appears to be a nasty grub/anaconda bug. I was upgrading my Fedora 16 laptop to Fedora 17 and ran into some issues.

This is the best I can tell what happened:

I installed Fedora 15 using the following partition scheme:

1G /boot partition 122G windows vista partition (NTFS) 50G Fat16 partition 526G lvm partition * swap, /, /home and /srv are each LVs on this VG

upgraded to Fedora 16 using the DVD.

upgraded to Fedora 17 using the DVD, anaconda had an error message stating there were "bootloader problems" and said there would be issues rebooting.

rebooted and got the grub rescue prompt.

rebooted again into rescue mode, using the fedora 17 DVD, and tried to do 'grub2-install /dev/sda' which is where I got the above error:

    Path `/boot/grub2` is not readable by GRUB on boot. Installation is
    impossible. Aborting.

After some googling I found this bug report but there didn't seem to be any resolution.

So I banged my head against this problem and stumbled into a work-around. (this should be done in a chrooted environment inside the rescue method of the Fedora DVD)

  • backup the /boot partition

    mkdir /boot-backup
    rsync -rav /boot/. /boot-backup/
  • unmount and zero out the boot partition

    umount /boot
    dd if=/dev/zero of=/dev/sda1 bs=1024k
  • create a new ext4 filesystem

    mkfs -t ext4 /dev/sda1
    mount /dev/sda1 /boot
  • re-run grub2-install

    grub2-install /dev/sda
  • copy the files out of the new /boot into /boot-backup

    rsync -rav /boot/. /boot-backup/.
  • copy the backed up files BACK into place

    rsync -rav /boot-backup/. /boot/.
  • re-generate the grub2 config

    grub2-mkconfig > /boot/grub/grub.cfg

at that point you should be able to exit the rescue mode, and reboot and things should be back to normal.

If not ping the bug and have them fix it.

Also, it's nigh on impossible to replicate by just installing Fedora 16 then trying to upgrade. It appears to be an issue with the Fedora 15 created partition scheme.

GoOSe Project Collaboration Manifesto

| No Comments | No TrackBacks

This document is a copy. You can always find the canonical copy here.

The GoOSe Project is only one of the upstream enterprise Linux rebuild projects. We are excited to work together with others in the EL rebuild space, and feel that there is plenty of room for separate but collaborative communities. The similarities in our goals should work for our mutual benefit. And the differences which exist should only strengthen us all.

So we need to answer this question:

In what can we collaborate?

What follows are the results of a pretty free-form brainstorming session we had at one of the GoOSe Project's weekly meetings.

  • Shared infrastructure

    Sharing certain pieces of infrastructure makes a lot of sense.

    • Upstream Sources Mirroring -- Although there are various places that one can find the upstream sources having another collaborative mirror for the various rebuild projects could be beneficial.
    • Cross Project Backups -- Doing backups is difficult, if we spread out the load of backing up our repositories, scripts, look aside caches, configuration files we could restore in the case of a catastrophic failure
    • Project Mirroring -- Distributing the final product of our efforts can be daunting as well. Again, being willing to mirror the ISOs or packages of the other projects will help make all the products better accessible.
    • Help Building and Configuring Non-Shared Infrastructure -- Most of these projects are using the same or similar tools, Koji for example. Working together on setting up these systems will help reduce the amount of constitutional knowledge(better phrase here?)
  • QA help

    • Building QA Test Suites -- Due to the common ancestor of these projects, a common set of test cases could be useful. This should be extensible/fork able for the test cases which would be specific to a certain project.
    • Cross-QA Checks -- A very good way of testing if something is correct is having someone else run the same test as you. Sharing the QA work and results would reduce the need for everyone to figure things like this out for themselves. I think the best example of this would be the information needed to verify ABI compatibility.
  • Documentation

    Of the many places where EL rebuild projects can easily collaborate, the best return of investment is sure to come from the documentation side. Again the common ancestor of these distribution works in our favor. There is a great need for documentation and how-to's. Sadly most good documentation is out of date. Thus a collaborative documentation effort would help lighten the load and keep things better up to date.

    Other documentation-like ideas include common wikis, a common askbot-style system, etc.

  • Distribution Creation Information

    The main product of all these different projects is a Linux distribution. Sharing the problems and various solutions to compile and create this product helps to make the entire community better. Things to share include:

    • package build order
    • package meta data
    • package content signatures
    • ABI compliance information
    • upstream trademark issues/problems
    • missing build dependencies
  • Community Enhancements

    As the number one goal of the GoOSe project is to build the community, we spend a lot of time thinking about the ways to enhance the ENTIRE community ecosystem. We plan on sharing what we learn for best practices for:

    • community management
    • meetings
    • organizational structure
    • updates to your community about your progress

    We also plan on working on spreading the word about our community goals. Some tools to help us accomplish this include:

    • EL Rebuild Planet -- Would others be interested in joining this?
    • Social Networking -- Twitter, Identica, Facebook, etc.

    We also want to practice what we preach about making this truly a community based idea. Please fork and update this document to help improve it.

With a stronger community around our projects we can all have a better time accomplishing our goals. And hopefully making the world a little bit better than we found it :).

Introducing the GoOSe Project

| No Comments | No TrackBacks

Introducing the GoOSe Project

Recently I blogged about how some friends and I were talking about what it would take to rebuild an enterprise Linux project. We started researching what we would need and we got started. The result? The GoOSe Project.

What is the GoOSe project? It is a community rebuild of an enterprise Linux distribution based in the USA. But it is also more than that. The GoOSe project is more importantly a practice in building an Open Source community. We wanted to practice what we preach about how best to build a community. It just so happens that the product that this practice produces is a Linux distro.

Why "GoOSe"?

The name of the project was just a bit of riffing done on IRC(look around 20:03.38). We have quite a bit done as far as getting things rebuilt and have some ideas as to how we want to run this community. We have an ethos. A Koji instance running our builds. And we are just about done with the Buildroot needed to make a fully functional system bootstrap.

The GoOSe project is "in progress" to getting the first build out. We are currently just about done with the first step which is to compile all the upstream packages into a buildroot we can use to compile our packages. A more detailed status update will be blogged about later.

Why another rebuild project?

As I mentioned above, community is the hard part. And although there are various other projects similar to ours, we think we can fill a certain niche.

The centOS project for example is probably the biggest rebuild project out there and it caters to those who are looking for a 100% drop-in replacement for the upstream Linux distribution. Scientific Linux is built as a base for supporting computing clusters and experiments.

The GoOSe project is not out to replace or out-do any of these projects. We are here to build a community of passionate folks to build the best community-based enterprise operating system around. We will be sharing everything that we learn, everything we modify, and everything we build to help use create this distribution. We all benefit from an improved community structure. We also benefit from having more than one project meeting different goals/niches. The things we learn also help all the other projects out there to have a well functioning system themselves.

GoOSe Goals

Our goal with building the GoOSe project is to have a cool community built around an easily constructed set of enterprise-grade software. The technical pieces of the equation taking the hard part out of building an enterprise operating system. Herlo said it best when he said this with regard to the vision of the project:

"[I see the GoOSe project as a] group of friends hanging out, building cool things with an enterprise-ready technology. Not exclusive to building the technology itself."

Come help us

We're already working away on these goals. You can come and help us. Here are some resources to get you started:

There is still plenty to be done. Come join us. It'll be fun I promise.

Nagios Config Template -- Goozbach rewind

| No Comments | No TrackBacks
Last week I got an email out of the blue.

it said:

While trying to organize a complete mess of an nagios install (half borked
by a nagiosql install) I ran across a post where you offered up an example
of your nagios configs. The link is now password protected, I know the
post is over 2 years old and things change....but I was wondering if you
still had that info available?

I did poke around your github but didn't find any nagios info.

This email was in reference to a comment I made to this blog post by the Standalone Sysadmin.

Well recently I re-did my webserver and the link mentioned in that comment has disappeared. So I took the time to drop the previously mentioned config template into a github repository.

I should mention that I've not done any Nagios work in a few years now, and moreover I only dumped the contents of the tarball mentioned in the original comment into the github repo without any changes or updates. So I don't know how well this would work with more recent Nagios configs.

If you find it useful, drop me a line.


Step 0 -- Build a Community

| No Comments | No TrackBacks

In a recent blogpost I discussed what would be needed to start a rebuild project based on sources provided by a prominent Linux distribution.

Some friends and I started working towards this goal. We quickly determined that as interesting and difficult the technical problems and hardware needs for such a venture are, they are surprisingly the easy part. Much more difficult is the task of building a community.

Community is Key

The key to any vibrant Open Source project is the community behind the project. Without a living, breathing community, any project is doomed to end up dead.

Analyzing other Open Source projects out there I discovered there are two distinct communities involved with any “successful” project. The contributor community and the user community. And yes one is a sub-set of the other (or ideally would be). These two communities are symbiotic as well.

Contributor Community

The contributor community is the core of a project. It consists of volunteers working on task relating to the creation, improvement, and release of a project. Without the contributor community the project could not exist. This isn’t to say the contributor community works in a vacuum. The contributor community relies on the user community for a few things:

  • First and foremost: Validation. Working on an Open Source project is a labor of love. Volunteers give a great deal of their time, talent, and work to an Open Source project. A vibrant user community validates the work of the contributors.

  • Second: User base. A project without users is a business without customers. No users; no project. The more users a project has the more willing contributors will be to work on the project.

  • Third: Feedback. There’s an old maxim: “Many hands make light work.” The Open Source equivalent to this is “Under many eyes, all bugs are shallow”. The larger the install base, the more likely quality bug reports will be generated and problems will be found and fixed.

User Community

The other half of the project exists in the user community. The user community consists of those who download, install and use the project in their daily work. The user community depends on the contributor community for few things as well.

  • Number one: A Quality Product. People are using large Open Source projects in the core infrastructure of their businesses. A quality product makes this possible.

  • Second: Transparency. Given the importance of a quality product. Users are eager for updates. These can be either security updates, new features, new releases, or just “we’re working on these problems”. When you develop your project “in the dark” users will start to get leery. If you miss big releases regularly they start to loose confidence. Fear of a dying project can lead users to find alternatives or to fork the project themselves.

  • Finally: Permanence. The user community depends on the contributor community for permanence. Open Source projects are living things and require constant care and feeding. They have their own life cycle. Open Source projects are created by volunteer forces. Those volunteers’ priorities change, they move on — for whatever reason. A completely open development model helps overcome these speed bumps that can affect a project. Some recent events come to mind with regard to the movement of a community:

Another item to think about is reducing the “bus factor”. The access to the entire project should not be limited to one member or small group of members of the contributor community. It should be available to all of the core trusted team. See CentOS Project Administrator Goes Missing.

The ultimate answer to all of these concerns is to make the entire project VERY easy to fork. Using new distributed SCM tools makes this pretty easy. The ideal Open Source project should be able to be completely forked in a weekend at the longest.

Community Trust

When the user community and contributor community trust each other, things really start to happen. More contributors show up to help as they feel valued and want to participate. The stronger contributor community means a higher quality, better trusted project. The higher quality project then attracts more users and the feedback cycle grows. The output of this feedback cycle is better support, and a better product.

Attracting Contributors

After defining what the communities are and what they want, let’s talk about how to attract more people to both. To attract people to the contributor community you have to have a very low barrier of entry for contributions. I’ve found one of the easiest ways to do this is with a tool which allows for easy contributions. One such tool is GitHub. The manner in which you contribute to a project on GitHub is easy:

  1. Fork the repository

  2. Make changes in your fork, and push the changes back to GitHub.

  3. Send a pull-request.

  4. The upstream project will accept or reject your change set.

This low barrier of entry for a contributor also works as a great apprentice method. As people show their competency through things like these pull requests, they can easily be apprenticed into the core group of contributors. This makes your project a true meritocracy. This apprentice system balances access with trust, and gives a clearly defined path from user to contributor.

In order for this apprentice system to work you need to have a list of todo items. Ideally this list would be full of low-hanging fruit for apprentice contributors. Things such as improving documentation, or adding an additional feature or simple bugfixes are good examples.

Advertising the hard problems your contributors are working on in a todo list or using blogging/social media give a potential contributor a good idea of what is in store with the core team. The fun part of being a contributor is to work out how to solve the difficult problems.

Attracting Users

The easiest way to attract users is to have a high quality product. Another important tool for attracting users is a vibrant, open collaborator community.

Regular releases are key. Users (rightly) see regular, on schedule releases as a sign of a healthy, mature project. Usually the first signs of a dying or dead project are the lack of regular updates, or delayed updates with no explanation.

Users expect communication from the contributor community. This communication is multi-pronged. Good bug reports are a key form of communication. Bugs should be “scrubbed” regularly looking for bugs which are show-stoppers and when a bug is fixed the “fix” and when it will be packaged should accompany every bug closing.

Other communication channels include the project’s documentation. Documentation is something which can lag behind very easily. Ideally the documentation for a project is auto-generated directly from the source code. If auto-generated documentation isn’t easy to do, then a documentation team should be part of the contributor group.

The project should also have a blog. This blog should entail interesting problems being faced, the current status, highlights of people who are working with/on the project. There should also be a planet for all the people who have an interest in your project to share their entries. To drum up traffic, the main blog could feature something from the user’s planet regularly.

Social media is another avenue for attracting users. Getting the news out on new releases, or other news items is easily done using the various social media sites.

Holding your meetings in a public place is another way of being transparent to your user community. Ideally these meetings would be in a medium such as IRC so there is automatically a transcript or log of the meeting minutes.

There should also be a status page where users can quickly get an idea where the project is with regards to its goals. Things to include here would be tasks needed to finish the next release, show stopper bugs/issues, automatic updates from the various tools used in the project, and any other information like that.

In Closing

A community is key to making an open source project work. Cultivating community should be the first priority of a project. A well groomed community spirit, will do more for a project than any technological advances or fancy tools any day. Remember the folks who would work on something need motivation. I for one would do it again for free.

Stay tuned for step 1 and finally an official announcement of what this lead up is about.

Broken Post

| No Comments | No TrackBacks

I just realized that my last post got eaten by the feed.

If you didn't see the sample files, please visit the post directly instead of reading it via a feed reader.

-- goozbach

Chain loading PXE servers

| No Comments | No TrackBacks

At my $DAYJOB we manage our internal infrastructure with Plan9. This means our DHCP server and TFTP server are both on the Plan9 system.

I really like to use Cobbler as an installation tool. However it works best when cobbler is in charge of PXE and TFTP.

To get around this problem, I'm using PXE chaining.


First step is to install the syslinux tree on the plan9 system. I did this by installing the latest syslinux RPM from here onto a Linux system (I used my Cobbler host).

I then created a tarball of the Syslinux tree by running this command:

    [root@cobbler ~]# cd /usr/share/
    [root@cobbler share]# tar -cf /tmp/syslinux.tar syslinux/

I then copied the tarball to the plan9 system and extracted it under the directory /386

PXE configuration.

The next step is to create the configuration which PXELinux will use. This is done on the Plan9 system, and it is relative to the file pxelinux.0, so I created the directory /386/syslinux/pxelinux.cfg/. Then I added the file /386/syslinux/pxelinux.cfg/default with the following contents:

This sets up our system so we may boot from PXE and have a menu which points us at two different OTHER PXE servers.

Plan9 /lib/ndb/local configuration.

The next step is to configure the Plan9 system so that when you boot your machine it will boot off the network and show the menu. I didn't setup my Plan9 environment so I don't know the details of making it a DHCP and DNS server, but this is the section of the config for a server which will be using the new PXE menu:

Usability Configs.

One last thing I did was to reduce the default timeout on the PXE configuration of the Cobbler server. On the Cobbler server, I edited the file /etc/cobbler/pxe/pxedefault.template, and changed the timeout from 200 to 100, which makes the total timeout equal twenty seconds (ten for the first menu, ten for the cobbler menu)

I hope others can find this information useful.

Bootstrapping a RHEL Clone

| 1 Comment | No TrackBacks

There was some interesting action on the CentOS mailing list recently which stemmed from a tweet posted by Dag Wieers.

The question posited is: "How does one rebuild RHEL in a timely, open and collaborative way?"

Well, a few of my UTOSC friends and I did a little brainstorming and came up with what we think a good bootstrapping infrastructure for just such a project would entail:


Dot File Management

I have always wanted a simple way of managing my bash file settings (.bashrc, .bash_profile and the like). I had built a system much like this one, but it didn't work very well. So I created a ground-up rewrite.

You can find the latest code for _Dot_Files on the project's github page.