Nathan’s Thoughts

Hiring: Operations Developer in Orem, UT

2010-09-07T14:53:00.000-07:00

[edit]

We hired two great people. Thanks for all the aplicants!

Getting started with tech.lds.org

2010-08-30T08:31:00.000-07:00

I discovered tech.lds.org the other day. Most of the software projects you can participate in on their site are Java projects. I dislike Java. Nevertheless, I'd like to participate in some of them. I'm having so much trouble getting the environment up and running that I thought I'd document the journey just in case it helps someone out in the future (or so I can redo it when I get a new machine someday).

[I'll keep updating the outline below until I get stuff working]

My system: OS X 10.6.4

1. (Piece of cake) I did the basic a) signing the contributors agreement, b) registering an LDS account, and c) sent an email to get an account on the bug tracker, as outlined on https://tech.lds.org/wiki/index.php/Requirements_for_Participation

2. I installed the LDS eclipse distribution

[post in progress...]

Kids grow up fast

2010-02-14T19:40:00.000-08:00

About two weeks ago, my son Adam started crawling a little bit. Now he's zooming around the room on all fours, and just today he managed to climb up the two-step stairs from the family room to the kitchen. Then he did it three more times in a row. (Sigh) They grow up so fast. Time to put up the gates again.

Keeping SSH sessions alive through a stateful firewall with ServerAliveInterval

2010-02-14T15:33:00.000-08:00

So with the redesign of our internal corporate network I've found myself one one side of a stateful firewall while all of the servers I SSH into for long periods of time are on the other side of the firewall. This mostly doesn't change anything, except for I found that I lose my SSH connections a lot. All the time, actually. The timeout appears to be somewhere around 20 minutes. Stateful firewalls actively track TCP connections, but they don't want to track them forever, so they kill connections that they think are inactive. Like my SSH sessions that I typically set up in the morning and expect to persist all day whether I use them or not.

I tried fiddling with the rules on the firewall to increase the state timeouts, but unfortunately there is no global setting for that on pfSense 1.2.3 that I can find, and trying to set that on all umpteen rules on all interfaces and remembering to continue to configure them is not a great solution. Besides, there are other firewalls out in the world that are not under my control.

The best solution that I found was the ServerAliveInterval and ServerAliveCountMax settings for the SSH client. It's simple to use. On your any-major-non-microsoft-operating-system box just create your config file if it doesn't exist already:

~/.ssh/config

And then use your favorite text editor to add these options:

ServerAliveInterval 60
ServerAliveCountMax 1440

ServerAliveInterval tells your SSH client to send an SSH keepalive packet every 60 seconds, which should be well under the timeout of any sane stateful firewall. The firewall ought to notice the packet and reset any state timers that it's using to keep watch on the connection. By default, your SSH client does not send these keepalive packets.

ServerAliveCountMax tells your client how many of these packets the other end can fail to receive (presumably because you've pulled your network cord, or turned off your wireless card) before your client shuts down the connection from your end. I set it to 1440, which is 24-hours worth of packets at once a minute. The cool thing about this is that unless the SSH server on the other end terminates your connection due to inactivity (they don't by default), or a stateful firewall or some other network device between you and your target shuts down your connection, you can turn off your network interface, then turn it back on without losing your SSH connection. Fun stuff.

Adam Nathaniel Stocks

2009-07-12T14:28:00.000-07:00

My son, Adam Nathaniel Stocks was born on July 9th, 2009 at 11:22pm. At birth he weighed 8 lbs. 7 oz. and was 19.5 inches long.

His delivery was great, no complications. Kate and Adam are both in excellent health. We are, however, getting a reintroduction to the wonders of sleep deprivation.

APB: Looking for a good developer to hire.

2009-06-09T21:28:00.000-07:00

Update (August): Thanks to everyone who applied! We hired a great guy who we expect will be very valuable to our team. :-)

Update (July 10th): Interview period is over. We had an excellent batch of candidates. I'll be contacting the best ones on Monday to schedule the second-round interviews.

Update (June 16th): Adjusted the salary upwards and massaged some of the wording.

-------------------------

Operations Developer

Job Summary: Software development and Operations.

Job Description:

Develop software to automate, replace, interface with, or assist in any or all of the following items: Keep all production corporate infrastructure secure and operational at office locations and datacenters. Corporate infrastructure includes computers (workstations, laptops, and servers), physical security systems, phones and phone systems, and network components. Help create requirements for internal products for which you will have production responsibility. Be involved in planning, creating, purchasing and putting into production new systems and software. Be involved in creating, testing, and implementing disaster recovery plans. Interface with employees and customers to determine needs, provide solutions, and troubleshoot problems. Accomplish any other duties as assigned by supervisor.

Requirements:

- 5+ years programming, software engineering/architecting, or equivalent experience
- Enthusiasm
- Aptitude
- Social skills
- Problem-solving skills
- In-depth experience with various non-Microsoft operating systems, esp. Linux
- Ability to handle pressure
- Your references have something positive to say about you

Pluses:

Please note that we don't expect applicants to have experience in all these areas, but it gives you a good idea of what the job deals with.

Programming and Scripting Languages
- Regular expressions (cross-language)
- Python (esp. Twisted)
- C/C++
- Perl
- Bash
- TCL
- Objective C / Cocoa
- Javascript
- HTML (okay, okay, it's a markup language)
- ...and others. The more the better. Except Java.

Education
- College degree (Computer Engineering, CS, IS, EE, etc.)
- Security certifications (CISSP, etc.)
- Other technical certifications
- Common sense
- Ability to spell words correctly
- Good grammar

Security & Security Software
- Good understanding of cryptographic principles
- iptables
- nmap
- nagios
- netcat
- Nessus
- Hydra
- OpenSSH
- OpenSSL
- SAINT
- tcpdump
- xprobe2

Non-work experience (with technical subject matter)
- Personal project portfolio
- Participation in open source projects
- Leadership experience
- Volunteer/Community service
- Cool and interesting hobbies

Operating Systems
- Linux (esp. Gentoo)
- OS X / OS X Server
- FreeBSD
- OpenSolaris
- Microsoft ones

Networking
- TCP/IP
- Ethernet
- CAT5/6 cable creation
- Fiber-optic cabling
- Managed switches
- Network design, performance, monitoring, and troubleshooting

Software
- asterisk
- apache
- dhcp
- dovecot
- emacs
- git
- ldap
- lighttpd
- PostgreSQL
- ntp
- postfix
- subversion
- vim
- X11 / XDMCP
- ZFS (the filesystem)

Hardware
- Server construction
- Rack installation
- Big UPS units
- A/C equipment
- Electrical circuits
- printers (esp. HP)
- Alarm systems
- VOIP converters
- routers
- telecom equipment
- projectors
- monitors
- speaker systems

Sleep!?

2009-03-15T20:26:00.000-07:00

Why is it so hard to get a good night sleep? Besides kids waking up in the middle of the night, I have the hardest time falling asleep. I can't take any significant sleeping medication because I'm on-call 24/7 (so if I get a call from work at 3:00am, I have to jump up and take care of it).

My inability to get to the sleepy state is great for pulling all-nighters when a server fails and not falling asleep on long car trips, but the rest of the time it is a huge obstacle to ever feeling rested.

Any suggestions?

A good quote from my primary lesson today.

2009-02-08T21:11:00.000-08:00

When the Lord commands, do it.

~ Joseph Smith

Where did Nathan go?

2009-02-08T19:58:00.001-08:00

I was doing so good. Then wham! I'm gone for awhile. What happened? I don't think anyone reads this thing, but since I'm mostly doing this silly blog for myself I might as well explain what happened.

Pretty much the opposite of the recession happened at work at the end of September. I work in computer security. In the financial sector. For some reason the big...organizations...are really getting serious stemming their losses from...sub-organizations...getting hacked. So one of those...organizations...decided they liked our company's security services enough that they sent 600,000...sub-organizations...notices that they had to come to our company and sign up for security services or be assessed a monthly fine.

Sounds great, no? It is! BUT, they didn't really think the numbers through and sent the notices out over 10 calendar days. Ya, 60,000 organizations per calendar day (more on Mondays? Mail's not delivered on Sundays...) get notices. We started getting over 5,000 calls per day. We started hiring, and hiring, and leasing new office space, and hiring, and leasing, and hiring, and provisioning. In 60 days we went from an ~80 employee company to a ~225 employee company. And I was in charge of getting the office space, provisioning the related network, phone, computer, server, etc. equipment for every single one of those new employees.

I've never worked so many 80-, 90-hour weeks in a row in my life. By Christmas things were pretty normalish again. But the company is different. We now occupy three buildings instead of one. I don't even know everyone's face any more. But business is good! That's a blessing in these times. We're providing a lot of employment for our local community too.

I have three extra months of projects that have piled up, though...(sigh). To handle it, my boss lent me a copy of "Get Things Done" and I bought OmniFocus both for my desktop and my new iPhone (!!!) to try to get things organized. Using an actual organizition tool, I discovered that (1) my list of to-do's is more humongous than I thought, (2) I spend 80-90% of my time on stuff that's not on my lists, and (3) the deadlines I set for myself for the big stuff on my list are completely unrealistic.

So, regarding #1, my list is stinking huge! The tools help with that. I've spent hours and hours putting to-do items into OmniFocus. I've got all the big items, and most of the little items. I have about 50 emails left in my inbox to go through (down from over 700!) and a few dozen bugzilla items to go through. Most of those will end up being small tasks to add to the lists.

Of #2, I've been training some of my Ops guys to take over the routine stuff that takes my time. Phone provisioning being the biggest one. The more I can train my guys and push responsibility for things down to them, the better it will be for the whole organization. I stop being a bottle-neck, and we also lower our bus-factor. I can actually work on the projects that make a strategic difference, instead of day-to-day-keep-the-car-running stuff. Not to mention that my guys become more valuable the more they learn! Eventually, I'll be able to start pushing the strategic stuff down to them as well. Unfortunately, there will never be a time when 100% of my time is spent on things on the list. Too much of my job consists of things that can't be foreseen (the phone server hard drive died after only two months?? 600,000 people are all calling us WHEN?) But I think it's reasonable to expect to be able to spend ~ 80% of my time on the list on normal days.

For #3, practice makes perfect! Now that I'm actually using a tool to set due dates on my projects, I can see whether I hit them or not and adjust my expectations. It helps when you're not trying to keep 300 projects and their due dates in your head.

Spore finally works (aka DRM = Epic Fail)

2008-09-28T14:03:00.000-07:00

After nearly three weeks of very slow back-and-forth with EA's online tech support, they finally indicated that they weren't going to do anything about the fact that my CD key had been pirated before I bought the physical game (either that or EA printed more than one of the same key). Either way, EA "helpful" support can be summed up in this line of theirs:

"The game is already register with account "tkth_yay@yahoo.com". Please login using this account and its password. (EARepVladimir)"

Ya, not my email. I told them so and they basically responded "too bad." So I went back to Wal-Mart where I bought it and exchanged it for a new one with a new key (because EA refused to close the other guy's account OR issue me a new key).

Then I spent nearly FOUR HOURS trying to find out how to change the license key on my installation without losing all my offline creations. Google failed. EA support failed. Forums finally worked.

I have not, do not, and will not buy anything other than Spore from EA while they keep up this draconian, non-functional DRM. Spore is the single exception, because there simply isn't another game like it. If there was, I wouldn't buy Spore either.

DRM = EPIC FAIL

Spore

2008-09-15T09:07:00.000-07:00

I bought spore. It's really fun, despite the fact that it runs very poorly on the mac. Even at 1024x768 with all the detail set to the lowest settings, there tends to be a lot of lag, visual artifacts, and crashes. But I'll stick it out, if only to cast my vote for games on the Mac (well, and it's still a fun game, and it doesn't crash as much if you restart it every hour or so).

Overloaded

2008-09-03T08:26:00.000-07:00

I've got a pile of projects from various sources that everybody seems to want done yesterday, plus a lot of loose strings from past projects that still need to be tied up that nobody else even wants to understand, let alone allow time for. In addition to that, the staff I manage has been decimated by the start of the new semester (college students), so I need to hire at least one more person to help pick up the slack there. I get up early in the mornings to exercise with my little brother-in-law. After work is family time and a struggle to get to bed at a decent hour. I get no lunch break half the time. I'm really tired. I think I'm overloaded.

Facebook

2008-08-31T20:28:00.000-07:00

My little-brother-in-law, Jonathan, is staying with us for a few months before he goes on his mission. He's not really into computers, except for Facebook. I've never really looked into the whole myspace/twitter/facebook/i'm-not-into-computers-but-i'm-a-social-butterfly thing, but he's so engrossed in it, I decided to try it out. So far, I'm impressed by the responsiveness of the site and the nice clean layout of their introduction pages. I've tried the linked-in thing, without any meaningfull results (though it may be useful for job searching if I ever need that sort of thing)--perhaps facebook will be more engaging.

It's all about the RAM

2008-08-08T12:56:00.000-07:00

At work lately I've been profiling five servers that we bought last fall. We spec'd them pretty high. They each have eight 2.66Ghz cores and 4GB RAM. We paid an extra ~$500-$1000 for those processors at the time, as compared to a 4-core setup, or an 8-core setup at lower clock speeds. The thinking was that we wanted them to be able to handle high loads.

Unfortunately, it turns out that the tasks that these servers perform are way more memory-intensive than CPU intensive. Since each task takes about an average of 4% of one CPU and an average max of 80MB RAM, we can do about 50 simultaneous tasks before we start hitting swap space, and we're only running about 20% overall processor usage.

But RAM is cheap! Today I went out and ordered 60GB of RAM (sixty gigabytes!) in the form of 30x2GB FB-DIMMS for less than $3,500 with tax and overnight shipping included. That's enough for each of the six RAM slots on each of the five servers to be filled, leaving each server with 12GB RAM.

On a related note, I went and got 4GB of RAM for my MacBook Pro for only $100. Love it!

Now if only I had somewhere to use 20x1GB FB-DIMMs...

iPod Nano Touch

2008-07-30T11:00:00.000-07:00

After reading this Macrumors post about someone's idea for a taller iPod nano, I said to myself "Oh give me a break. They're not going to copy the Zune. If anything, they'll go more innovative like they have with the rest of the iPods." What does that mean? iPod nano touch. So here's the mockup. (You should really click on the picture to see it in full size!)

(Feb 2009) Update: Ya, I was wrong. They went with the taller version of the nano like the rumor said. BUT! The screen rotates when you turn it horizontal. Looks sorta like my mockup with a dial on the side. :-) Oh well, I wouldn't have bought a nano touch anyway -- I love my new iPhone too much!

Duncan goes free...

2008-07-29T19:09:00.000-07:00

Duncan McGreggor, whom I met up with at OSCON 2008, has quit his job at Divmod and is going camping in the Rocky Mountains for awhile. Good luck, Duncan! Duncan was COO at Divmod, and heavily involved in Twisted.

The Hard Drive is Dead -- Long Live the Solid-State Drive!

2008-07-28T19:35:00.000-07:00

I've been telling my friends for some months now that hard drives are on the way out, to be replaced by solid-state drives (SSDs). I've had no doubts about if that would happen, just when. It's time. The moment I saw this article about the announcement of 128GB and 256GB SSDs, I officially considered hard drives as dead.

Home Videos

2008-07-27T18:54:00.000-07:00

I've really gotten into working on Home Videos lately. Each Sunday, I've worked on getting more home videos off of tape, processed with Final Cut Express + LiveType, and then put on a DVD with iDVD. Over the course of 3 years, several revisions of most of the software and one complete OS revision, the hardest part has always, always been iDVD. At this point, I no longer customize anything with iDVD. I simply use the default theme, import the movie file from FCE, and cross my fingers. I find that I can get a lot done if I ignore iDVD.

Cocoa

2008-07-26T20:46:00.000-07:00

I'm working through Aaron Hillegass's Cocoa Programming for Mac OS X, Third Edition. I went through Chapter 13 tonight (making preference panels show up - yay!). Cocoa is extremely interesting. On the one hand, it's still annoyingly low-level (since it's C), and yet it's remarkably high-level (since it's Objective C and has a ton of huge libraries) while still allowing you to shoot yourself in the foot, C-style [dang pointers]. It's still a bit weird to have an error crash your code and not give you a descriptive traceback indicating the line that the error occurred on and a best guess of what went wrong. Instead, a gdb window pops up with the assembly command that crashed. Makes you appreciate Python and TCL a bit more...

I've heard many complaints about Xcode from Java people who have super fancy IDE's. My usual "IDE" is called "Emacs," so I really haven't seen anything to complain about. Xcode seems huge and fancy and featureful to me so far, compared to emacs. The fact that many of the emacs keyboard shortcuts work in Xcode also helps.

Portland

2008-07-25T20:57:00.000-07:00

Portland was interesting. There were lots of trees -- lots of evergreen trees, in fact. I had heard that Oregon was very green, but it didn't occur to me that so much of the green would be needles! It was in the 70's with nice weather the whole time I was there. I was interesting to return to Utah where a dust storm was kicking up with 95-degree winds at 8:00pm...which isn't that typical of Utah, but not surprising for nearly-August.

For everybody I met at OSCON this year (Mike, John, Duncan, Miguel, speakers, vendor-people) -- thanks for making it a great time!

OSCON Impressions (Friday)

2008-07-25T09:40:00.000-07:00

Tim Bray - A medley of languages?
- Lots of programming language usage graphs
- Did an interesting "raise of hands" survey of "what are you using now?" vs. "would you in an ideal world?" for a bunch of languages. Only Python and Ruby had the same or more number of hands for the "ideal world" part.
- A bit of light PHP bashing (yes you can use it, but boy the code tends to be lousy)
- Some heavier Java bashing, but a nod to it being big and huge and widely used in enterprise
- "C is flawless" if you want a language that represents actual machine code. Points out that pretty much all other languages and many big projects are written in C.
- OCaml is really fast.
- Erlang is really robust and parallel, and very recursive and hard to wrap your brain around.
- Scala is...a Java...something. Used by twitter.
- Fan runs on JVM and .Net...

Jeremy Ruston, head of open source innovation at British Telecom
Learning from Airports
- An airport represents a multi-sided, open platform.

single sign on
participants trust the platform (shops will accept your boarding pass as id)
Common standards for docking airplanes
Common signs in airplanes (to find the bathroom, for example)

- open source team tries to be very open and transparent
- try to share everything that they do

Q & A for all the above speakers
Very boring. All questions were hostile and aimed at the microsoft guy.

Clutter, an OpenGL GUI toolkit
- runs on both OpenGL and OpenGL ES - with layer called COGL.
- intended for mobile devices
- not intended for games
- manipulate 2d surfaces in 3d space
- discussions of "actors" - the basic building block in COGL
- you put actors in "scenegraphs" as children. Children can have children too. Whenever you manipulate something, its children are manipulated the same way
- container - more structured way of defining child relationships
- quick demo of using clutter in python

Damian Conway, The Twilight Perl
- starts out with a bunch of jokes about how perl is complicated and you can do tons of stuff
You can't (but he shows examples of how to do all of the supposedly impossible stuff)

have comments and code before the shebang line
leave the sigil off a variable
go wrong using fat commas instead of a plain one
access the $var outside of the "for my $var" loop
leave the parens off a for loop
have a qw without the qw
many more examples....but I'm not even a perl user, so I'm going to stop here.

I'm so glad I don't use perl! Sheesh. Very funny, though. Damien's a great presenter.

That's it for OSCON 2008!

txpyglet

2008-07-24T22:15:00.000-07:00

Some people expressed interest in my (brain-dead) method of integrating pyglet and twisted. I ripped out the most generic parts of my in-house commercial app and dubbed it "txpyglet."

Here's a little demo of it (blogspot shrunk it, get the "big" 5MB version here)

Here's the real meat. In pyglet, you simply subclass pyglet.window.Window and give it a pyglet 1.0-style run() method. When your app actually starts, use twisted's coiterate() method to coiterate with that run() method. That's pretty much it. It's clearer in the code. Let me say right now: Yes, I know pyglet blocks twisted for an indeterminate amount of time each time control is passed to it. It still works for us simpletons :)

You can downlod txpyglet here.

OSCON Impressions (Thursday)

2008-07-24T17:59:00.000-07:00

Thursday was crazy. Work kept interrupting, and actually made me miss one session block entirely. I struck out at all my session choices (boooring) except the morning "State of the Lightning" talks.

More notable, I met up with Duncan McGreggor (oubiwann) of Twisted fame. We had a good chat and missed Mark Shuttleworth's "software development methods" session because we were one minute late and they put him in the smallest room in the whole convention center. It had a capacity of around 30 seats give or take a dozen. Sheesh. And yet they gave some awful speakers with boring subjects huge 500-seat ballrooms. Figures.

I showed Duncan my twisted+pyglet project, and he talked it up in #twisted, and some folks actually sounded interested. We'll see if I can't get a post in about that.

There was one other thing worth mentioning. A really profane guy from New Zealand (whose name I missed) paradoxically gave a great presentation about teaching kids programming skills. It boiled down to:

Scratch is a cool little kids-programming-environment that lets kids script things
Lecturing kids doesn't work partially because they have 2-minute attention spans
Pair programming (you+kid) can work
Follow the interest of the kids
Typing is tough for kids
Math is tough for kids
Teachers need teaching about computers in many (most?) cases
You should volunteer your time at schools to help with this stuff. Don't expect pay. The work is the reward.

OSCON Impressions (Wednesday)

2008-07-24T17:45:00.000-07:00

Wow. It's really neat to see so many people in person that I've only ever read about before! Tim O'Reilly, Larry Wall, Keith Packard, etc. Here's my impressions:

Tim O'Reilly's Keynote

Boring for the first part (sorry, Tim). The interview of the two MySQL guys was much more interesting. A lady gave a 'no secret software for public voting/sensing/etc. hardware' presentation which was decent. I agree.

Keith Packard - state of X.org

They're making tons of progress (finally!), but boy X is still a mess. I stopped using Linux on the desktop in 2004, but it's nice to see them actually making progress after the whole XFree86 forking thing.

Josh Berkus - PostgreSQL, how to secure your data - whole picture.

Discussion of security in depth. Airplane analogies (perimeter-security-only is stupid). You need multilayer security. His threat vectors are worth noting:

Four Primary Threat Vectors:
1. sql injection
2. direct connection
3. webserver compromise
4. staff access

Personally, I'd put "direct connection" at the bottom. In my experience scanning our customers, they rarely make that mistake any more.

The most interesting part was when he went over how he often makes shadow tables in a separate "schema" that regular db users can't access, and then he adds triggers to all the regular tables so that on INSERT or UPDATES the row is copied to the shadow table first along with a timestamp and the db user. Nice...instant audit log! I need to try that at work on some tables...

Alex Martelli - Code Reviews for Fun and Profit

This presentation was quite a bit more interesting than I expected. He went through about a thousand points, of which I only randomly remembered to type in:

- always review code, like brushing teeth
- there should be someone who's supposed to review all code for some project/area
- don't waste human time on computer tasks ("no tabs in source code", "max line length", etc.)
- code should be clear and readable
- don't repeat yourself
- use meaningful and consistent names
- appropriate info for errors and logs
- check corner cases (test rare error conditions, exceptions, etc.)
- check for memory leaks
- check for security issues
- use version control
- review the diff before you commit, if possible
- don't review for longer than 60-90 minutes at a time, because after that time your reviewing quality plummets

Stewart Smith, MySQL
common mistakes that lead to data loss
- did we open/write/close successfully.
- not closing the file
- write/close/rename = FAIL --- write/fsync/close/rename = win
- fsync() on os x doesn't sync! What??? checking on this...

Ben Ramsey, memcached
- lots of discussion of how memcached works
- memcached is slower than a direct connection from one web server to one database server
- memcached is much slower when multiple web servers and multiple database servers are involved.

Update:
I forgot to mention that at lunch, Miguel de Icaza and two of his friends[1] sat down at my table and ate lunch with me. He's a funny guy. He introduced himself as "the guy who made Midnight Commander" and was constantly bantering back and forth with his friends. He ranted a bit about how he's doesn't like his iPhone 3G because it doesn't have email search or physical buttons, so he's switching back to a blackberry. It was fun to finally meet Miguel in person. If you don't know, he started a couple of tiny, little, obscure projects like Gnome, Gnumeric, Mono, Ximian [the company--acquired by Novell].

[1] Neil? from Novell in Munich, Germany [but not German], and...Amy? from Coverity. Someone correct me, I don't think I got their names right. Also a nice guy named Jack sat at the table with us, but I don't know anything else about him.

Third Time's the Charm?

2008-07-24T17:41:00.000-07:00

The biggest obstacle between me and blogging has always been the mechanics of actually constructing the blog (or choosing, installing, and using some program to do it for me). Maybe if I just let Google's blogger to the heavy lifting, I'll actually get this to work.

On to obstacle #2 -- finding the time!