Breaking a 2-year record

I arrived a full hour early for the interview. I had forgotten that the small campus of this potential employer was only thirty minutes or so from downtown Salt Lake City. I sat in my car, listened to the Laugh USA channel on my XM Radio, fired up my laptop, did some geeky stuff with a Linux kernel (a diff in order to see if I could back-port a USB definition for my Tungsten C, but no luck), read some of my Linux Journal, and waited until there were 30 minutes before my interview.

I sauntered into the building.

The signs in back looked nothing like they’d looked the last time I was here. Oops, wrong suite, one more down!

I walked in, talked to the nice older receptionist by the name of Carol, she handed me an application, and I Sat down to fill it out. I noticed a thirty-something guy filling out an application at the counter. He appeared to be another candidate. He was smartly dressed in a gray business suit, sported a shaved head, was lean, and had that kind of rugged, chiseled, outdoors-y look to him.

I glanced down at my small, but steadily advancing, pot-belly. No tie. Some business-casual slacks, long-sleeved red shirt, and black leather jacket. Damn, I am such a geek. I keep thinking I need to hop back onto my exercise regimen and get back the glimpse of six-pack that I’d almost had a year ago, but it keeps being pushed back for things that are more fun.

“It takes all kinds”, I thought to myself. In this kind of tech company, the contents of one’s brain was much more important than the fat content of one’s abdoment. I looked around at the people moving in and out of the entrance, and in typical “technology company” fashion, there were all manners of dress: ties, dresses, business suits, jeans with sweatshirts and sneakers. You name it. Good, I really wasn’t too far out of the norm. However, I ran out to my car and fetched a tie so that I was slightly more presentable. You never know.

I finished filling out the application. I really wish more companies would move to electronic methods for this stuff. My hand always cramps up because I write by hand so rarely. Give me a typewriter, and everybody will have a much easier time reading the darn stuff. Not to mention, it would be much easier for me to write it.

A few minutes later, I watched the interviewer, Rick, come up and introduce himself to the other candidate. “I’m Rick Soandso. Right this way.” He escorted Chiseled-Feature-Man down a short hallway and disappeared into what seemed to be a small conference room. I intentionally sat with my back to the short hallway, so that I would not see who was going in and out. I’ve found that things go most smoothly if I introduce myself for the first time having not actually seen someone, rather than having watched them float back and forth while staring at them repeatedly.

The first interviewer reappeared about forty minutes later, ten minutes after my scheduled appointment time, and introduced himself to me. He explained he was running a little late, and thanked me for my patience as he explained he had a bit more to do before seeing me.

A few minutes later, the other candidate emerged from the short hallway. He futzed with some business cards on the receptionist’s desk. When Rick showed up again and made a formal introduction with me, he asked Rick for a business card. Rick asked Chiseled-Feature-Man to hang around for a few minutes, and he’d provide him one.

Chiseled-Feature-Man (CFM) looked nervous. He was extremely self-possessed going in, and yet came out looking very anxious. Not a good sign.

Rick and I made small talk for a few minutes, after a somewhat less-than-graceful half-joke on my part about competition for job positions. He explained the company’s organization, what the role was of the position I was applying for, and some of the technology in use while caveating, “I am a manager, not a technical person, so I may not know what you are talking about”.

Eventually, the next interviewer was summoned. Brett, his name was. We chit-chatted about some of the same topics covered with Rick, then he began asking some historical questions about my background, mixed with a few technical questions. We chatted about Linux and technology for about 30 minutes, before summoning the final interviewer.

“Nate’s interview may be more interesting,” Brett informed me. “He has the legendary laptop.”

Now, I’d heard about this laptop. Rick had mentioned it to me, as how they evaluate the basic technical capabilities of candidates. I was a little nervous.

Nate entered, carrying the notorious machine. We spoke for about 5 minutes, covering much of the same ground again regarding my background and interests. I learned a little bit about him too.

“So this is the legendary laptop,” I segued, “why don’t we dive in?”

“OK,” he responded, shutting the unit down. “It has a problem with X. You don’t know the root password. See if you can fix it.”

I powered the machine up. After the usual BIOS screens, I saw the grub prompt (grub is a boot-loader regularly used on GNU/Linux systems), pressed “a” to append to the kernel execution command, added “single” to the kernel arguments so that it booted to single-user mode, and booted.

It went straight to X11, flashing an NVIDIA logo at me repeatedly. “Well, I thought you were going to single user,” Nate began, “but this is the problem. I think the last guy really messed it up.”

Since it was obviously stuck in an infinite loop (darn Redhat, and their policy of setting X to “restart” via init in runlevel 5, rather than “once” like it is on Gentoo), I powered the machine off, made a joke about hoping we were running a journaled filesystem, and then powered it back up again.

“Let me give you a little help, because this isn’t the problem you’re trying to solve. Add ‘init=/bin/bash’ to the end of the kernel invocation argument.”

I did so. Within a few moments, it reported a kernel panic and crashed.

“I think I need to find another laptop,” Nate informed me, then bustled out. He returned a few minutes later with another box, booted it up, typed in a single command, and then powered it off again. “Try again,” he asked.

I booted to single-user mode, changed the root password, added a nonprivileged account, gave it sudo privileges, went to init 3, and logged in as the unprivileged user. I like to have a log of the commands I run as root, and “sudo” provides this. I use it all the time on the laptop I’m using to write this blog entry.

I went through several things. I tried a “find / -atime 1 >~/find.log” to see all the files changed in the last day. I chatted with Nate for a few minutes while the hard drive churned. I opened another virtual terminal to check the status of the log, and it had found nothing. I realized later that I’d provided the wrong flags to “find” to actually have it find which files had changed in the last day. Oops.

Anyway, I tried starting X and redirected stderr and stdout to log files. “startx 2>error.log 1>out.log”

It hung. I hard-rebooted the system since I couldn’t get to a virtual terminal.

“This is an evil little problem,” Nate informed me. “Nobody in the last two years has been able to fix it in an interview.”

The machine rebooted. I again went to single-user mode, set the default runlevel to 3 instead of 5 (console mode, rather than X), and checked my logs.

“Could not find default font: fixed” the error log from X reported to me.

“Huh,” I thought, “I wonder if this uses a font server?” I don’t use a font server on Gentoo, preferring to specify the fonts in the X server configuration file since I get a small speed boost from that. On my aging 366MHz Pentium 3 laptop, every speed boost I can get, I use.

I checked /etc/rc5.d. Sure enough, the X Font Server wasn’t set to run. I checked if it was running (ps -ef | grep xfs), and it wasn’t. I changed the name of the symlink from “K10xfs” to “S10xfs”. I fired it up with “./S10xfs start”. I changed the link in /etc/rc3.d as well, just to cover my bases in case I tried to start X from init runlevel 3.

I typed “startx”.

It worked.

I said, “Looks like it’s working!”

Nate added, “Well, you’re not quite done.”

I removed the unprivileged user account I’d created, along with his password and group allocation. I cleaned up the sudoers file so he wasn’t in there anymore. I rebooted, made sure X came up, and ran “chkconfig –list xfs”, corrected it so it ran in runlevels 3, 4, and 5 with “chkconfig –level 345 xfs on”, and then Nate said, “NOW you’re done.

“I think, actually, one other person has managed to fix it in the last two years. And it used to be harder; there was a change to the font server that eventually Redhat fixed.”

I smiled. I realized with satisfaction that I’d won a round of the interviews. And I thought Nate was a pretty cool guy and had enjoyed his interesting, but somewhat tricky, problem. I think I’ll like working with him.

This is a company I’d love to work for. They have a solid reputation in the Linux community, and are working on some seriously cool cutting-edge stuff. They’re a startup, so there’s some inherent risk involved, but if they can meet my modest salary requirement, I’m certain that we can get along.

I called the recruiter on the way home, and left him a voicemail to inform him that the interviews went well. I think I was slightly overconfident, assuring him that I think I’d like to work there, and I’m certain my interviewers would like to work with me. Darn. I should keep that tendency toward overconfidence in check.

Regardless, I feel like I scored a home run. What a great feeling after my disaster of an interview a few days ago. Very cool stuff.

It’s not glamorous, but it’d be a good job with a company with a solid rep. That sounds like a lot of fun to me.

— Matthew P. Barnson – – – – Thought for the moment: Alcoholics Anonymous is when you get to drink under someone else’s name.