Doing what I do best

Finding the bugs that piss people off.  Yes, that is what I do.

After mucking around in TCL for several weeks (and seeing how horrid it was - at least the built-in version for these fpga development systems), I have a minimal tcl script that initializes the platform and then takes user input to change the fpga settings on the fly or read registers on the fpga for status and data connectivity.

The real automation is in python.  In only my third python script ever, I have it controlling the tcl system fairly extensively.  So far, it has helped find several bugs in the fpga development environment (like sudden and unexplained crashes), bugs in the fpga register behavior (why does reading the setting in one register erase all other data?), and even possibly other bugs in our pre-release development system (not to worry, this is why it is pre-release).  I still need to figure out how to recover when the tcl app crashes (currently it takes the python script down with it).   Easy enough to restart where it left off, but annoying.

The cool thing about this scripting (at least from the team's point of view) is how much it has helped speed up their work.  Before, to change one setting (knob) in the FPGA would take as much as 20 manual operations.  Spread that across 4 different knobs (each having 4-16 different settings) and 24 different data lanes, and that is a lot of manual tweaking.  My script so far loops through all permutations in about an hour, reading a control register over 1000 times per setting to ensure low error rates, then listing the best combinations.  Kind of like tuning a surround sound stereo for the best audio experience by tweaking the volume, bass, treble, and distortion on each individual speaker.  This is still only scratching the surface of what I will be developing, but it is a start.


As to the bugs pissing people off, they aren't mad at me for finding them.  It is after all what I was hired to do. 

And I am very good at it.

There and back again.

Life appears to be full of little twists and turns.  Four years ago, I finished my last contract at Intel, then spend the next few months finishing my degree.  The week I graduated, I accepted a job with Canonical, initially doing QA testing on netbooks (for the same team I worked with while in my last Intel contract), then switching to ARM testing when our team spearheaded that effort for Ubuntu.  That began best job I have ever had.  Then it ended, rather abruptly.  Largely due to market shifts and stabilization of the ARM port (our team was disbanded and moved into other platform teams, I just didn't find a spot to land in).

Less than two weeks after being let go from Canonical, where do I find myself?  Back at Intel.  And (as seems to be a running theme here), I found myself in way over my head.  Like treading water with lead boots on.  To say it looked dire is...well...putting it mildly.

My new job involves writing automation scripts in TCL to program and monitor FPGAs.  On Windows 7.  I will spare you readers of the gory details (largely because I am still wrapping my remaining brain cells around them myself).  The first 7 days, I seriously wondered why I was picked for this particular position (and why I accepted).  It wasn't like I didn't have other offers (average of 10 a week for Linux jobs).  This was way out of my comfort zone.

To start, I knew next to nothing about TCL (I'd heard about it, nothing more).  So, Monday I bought a book (Barnes & Noble rocks - shameless free plug).  By today (Friday), I have figured out enough of the idiosyncrasies of the language to get my first script working (read FPGA state, reprogram, check for errors).  The next step will be to make this script easier to interact with so it can be called from a master program on the fly.

I could have been much further along, but the first 1.5 weeks were largely spent with getting basic necessities (accounts, lab access, cube, still waiting for a standard issue laptop, etc).  The funny part is that most of the information in the system about me was based on my last contract, so manager approvals were misrouted.  Oops.




I still feel like I am in over my head, but at least I now have a rock to stand on.  I am really starting to feel like I can do this job, and do it well.

But I still hate Windows.  Just saying.

What does three up and three down mean?

Well to some, it is the end of an inning.

In my case, it is time to move on.

In my job, I was doing;
Only QA tester on Arm desktop,
Broke ground on QA testing on Arm Server,
Interacted as both tech support and community liaison with multiple communities of Arm Linux developers,
Automated Arm network installation and SRU testing,
and many other day-to-day activities.

I also would root cause as many bugs as I possibly could, often replicating them on x86 and amd64.  I even fixed a few bugs along the way.

But, due to shifts in work and internal reorganizations, I am no longer needed.  Yes, today is my last day here (well, maybe not on this blog).  I only recently opened up my resume to job hunters, and the calls are coming in rapidly.  I hope to land somewhere soon, possibly as soon as next week.  Seems Linux QA people are in high demand still.

But what about all my equipment?  What about the dedicated tower of Pandas?  For now, they are now busy searching for ET.  I might have future work for them next month.  They seem to be able to perform quite well, even keeping pace with the SS Itanic I have sitting here.  So what if they aren't as powerful as the multi-core 64 bit laptop warmers that most people have.  I'll bet they draw less power than the Speak & Spell that ET uses to make long distance calls w/o a carrier network plan.  And they support bluetooth.

So, I leave you readers with this:  "You have entered a dark place.  You are likely to be eaten by a Grue!"

It's alive!

Since my last post, I have been overly busy with test automation, fixing bugs, learning python (able to debug a lot of code issues, still need to start my own code projects), etc.  It has been shear and utter chaos.

Part of this has been lab expansion.  I am now up to 8 Pandas online, plus additional hardware (some of which I can't comment on, but it is COOL!).  In preparation of arm servers, I am in the process of gathering data on power consumption while running tests.  This involves a really nice Fluke digital multi-meter with data acquisition.  To simplify testing, I am using a Panda as a basis, as the hardware is now well known, and I can easily put the meter in between the power supply and the panda. 

Wires, lots of wires.

This is a desk test with an old beagle that I currently don't use for testing.  Figured if I was going to fry a test board, make sure it isn't in use.

So, with the wiring working, the next was to run a semi-meaningful load test.  Since this is ultimately for server testing, I wrote up (with the help of others) a simple lamp stack test that installs apache2, php, and mysql, then loads & verifies mysql with a large amount of dummy data, which php will query from.  Once that is setup, the test runs apache bench for a few iterations before really clobbering it with siege.  It took some fine tuning to scale the test down to run under 1 hour (first try ran all weekend before I manually killed it - oops).  Below is the graph of the data from the power meter, starting from idle.

The first bunch of noise (0-600s) is  apt-get install lamp-server^ installing the packages, followed by a slight increase as the database is populated and verified.  The spike from 750-1100 is apache bench.  The rest is siege.  The system is a panda running headless with a 16G SSD on usb sata, powered by the panda.  Everything was run locally. 

Note:  This is not intended to be used for comparison, only a system I have available.  No other systems have been tested, and I'm not even sure the tests I ran can be considered valid given the setup.  But for those interested in how much power their cell phones will consume when acting as a lamp server, here you go.  Enjoy.

Not bad for a first run.  And fairly easy with a 2 wire power source.  Now for the fun part.  To do meaningful power benchmarking on any platform, you really need to be between the power supply and the platform.  Testing at the wall is near meaningless.  The kill-a-watt I have on the wall jack behind my rack cabinet shows I am only using ~5 amps, and that is with 4 low power systems running full time.  Another one on a Core2Duo open test system (just a motherboard, video card, and drives - no chassis) shows ~.45 amps between PS & wall when under test.  It drops insignificantly when idle.   So, to properly test, I need to sit between the power supply and the board.  Now, the trick is to not overload the 10A max load on the meter.  AC, no problem, but my 450W atx power supply says  it will handle 22 amps on 3.3v, 15 amps on 5V, and 30 amps combined on 12V.  Time to break out a cheap power meter (in case of smoke).

This is going to be...{fun, expensive, deadly, ____}.

We will see.

Entering the Black Arts of Automation

Well, this is fun.  With all of the testing I do now on Ubuntu Arm, it is time to look at automating as much as possible.  The "fun" part is that I haven't done much coding or scripting since 2001 (When Perl ruled the day, and Python was barely in existence).  The other part of the problem (for me anyways) is that most of the existing automation tools we currently use in x86/amd64 testing rely on tools that just don't work on existing arm platforms (kvm/libvirt, kexec, ipmi to name a few).  So existing scripts that do things like, say, "reimage a platform" need to be done from scratch. 

And I am torn between using the antiquated tools I know (or remember using in the past), or learning a bunch of new tools.  And the overwhelming "helpful" responses I get when asking for some help in certain areas usually are "Use this tool, it is easy."  Of course, those people are well versed in that tool.  Try telling someone that only knows how to edit in vi how to use latex.  Yea, not going to happen.  Don't get me wrong, I am all for learning new tricks.  But I can't justify the days it takes to learn how to do one task that someone who knows how can whip together in a few minutes. 

Also, I learn by books and examples.  I've already spent a lot of money updating my book library (2 Python books, XML, Expect, and a few others).  My 4x8 foot bookshelf is starting to sag under the weight of all the books I have accumulated over the years.  And, no, I don't like ebooks for this.  I just bought the Exploring Expect book for my Nook Color, and while the information is proving very helpful, it isn't as easy to bounce between sections like a good print edition.

But I am making very good progress.  It used to take me 2-3 days to fully test each kernel SRU update cycle (4 platforms across 3 releases), most of it hands-on.  Now, I can (almost) start a full test sequence with the click of the mouse and check the results in a day or two.  There are still a few kinks to work out (like automating the preseed configuration and monitoring the reimaging progress), but we're getting there.  Other parts of this testing were outside of my control (tests that fail because of configuration differences between arm and x86/amd64 kernels for example), but they too are being resolved.  I have also hit an issue in the past where an SRU kernel had an update from the vendor that disabled video on my test systems (well, disabled HDMI for LCD port - which I don't have hardware to test).  Had that kernel gone to the general public after testing only on an automated, headless environment....

Once I have the SRU process fully automated, I can focus on automating other jobs.  They should be fairly straight-forward, as the core work (reimaging) will be done, and I can just launch a job to install & run packages at will.

The other big (bigger) problem is (drum roll), infrastructure.  When I started on this route, I had 4 systems.  I now have 15.  Some of this can be deployed in our QA lab, mainly the headless stuff.  I'm not sure I could justify the expense of equiping the lab to be able to do remote desktop testing (KVM/IP for HDMI is expensive, and then there is audio, bluetooth, etc).  Server stuff yes.  Add to that the power relay I have (see earlier blog) works fantastic...on 4 systems.  It is expandable up to 256 relays, but my personal budget...well...

Some people also think I should focus on automating the desktop testing somewhat.  Well; 1) The interfaces enabling any type of desktop automation is currently broken in gtk3, and 2) The desktop changes too rapidly to automate (gnome 2 -> netbook-launcher ->Unity/Unity2D in 4 cycles).  That means scrapping/rewriting a lot of tests every cycle.  Much better to get tests for server/core stuff running now, and hit desktop later.  There is a lot that can be tested at that level, that desktop will also benefit from.


Hmm, I wonder if this blog could be automated.

The only constant in this equation is change.

With winter rolling in (practically overnight), it is time to focus my spare time on inside-the-house projects. I am working on remodeling our 1934 2 story (plus full basement) house; rewiring to current code, new plumbing, insulation, and drywall. First on my project list for this winter is my office. The ceiling fan/light is on the shared knob & tube circuit as the rest of the lights in the house.   And the single wall outlet in the room is shared with an outside plug with a grounding wire that is almost invisible (and to think I had as many as 15 computers running on this). There is almost no insulation, so during the last year, temperature in this room went from 65F (18C) in the winter with the furnace running, to 85F (30C) in the summer with air conditioning. The problem is that my office is an addition on the south end of the house, so it gets sun all day long (when the sun is out). It also doesn't have a full basement, only a crawl space with a large opening next to the stairs to the back door.

In years past, I could regulate the temperature better by turning on or off computers, but since I am now doing ARM based testing, my systems don't generate nearly the same amount of heat. My tower of 4 Pandas with drives consumes less than 20 watts of power. For comparison, the average x86 processor today uses 65 watts, and that is just the processor. Think in terms of light bulbs. A 60 watt bulb generates enough heat to burn your fingers if you try to remove it while it is on, whereas a 20 watt bulb is only slightly warm.

Back to this relocation project, I have moved all of my test systems and their 8' table to a big room in the basement, and it is now back online after reimaging my serial console (serial-killer) with 11.10 server. The rest of the systems are booting fine and waiting for me to start slamming them with tasks. Now I just need to move my other 8' table downstairs, along with my desktop system and netbooks. This brings it's own set of problems, as my current firewall is also here along with the dsl-modem. The firewall is an old Pentium III-450mhz (clocked to 300mhz for passive cooling) running Mandriva 9.0 (2.4 kernel and just plain ancient). It has been my firewall now since ~1999 (well before Ubuntu existed). I reinstalled it in 2003 when I moved, mainly because the 10G drive in it had failed. It has been running since then, with the only downtimes due to power outages. If it isn't broken ...

I have a new firewall system in a 1U rack case currently running in the basement. This system is based on a Pentium-M 800mhz with dual gigabit ethernet ports. and using an 8G CF card on an IDE adapter for the OS. It is currently running Ubuntu server and manages DNS and DHCP for the house. It will also provide IPv6 control once it is the primary firewall. My only reservation is the downtime that will be inflicted when I move the dsl-router. When I built and configured my old firewall, it was setup such that it looks like a hole at that ip address, neither responding or timing out (unauthorized access is put into a holding pattern instead of being dropped, a trick I learned during the Code Red virus days of 2001). I am having to relearn how to setup a firewall though, as a lot has changed since 2003 (like say the kernel). I also had to run new phone wiring as the existing wires are the old style 4 wire phone lines, all terminating at a ceramic block with 2 brass studs and nuts in the basement. Talk about scary. With any luck, I should have the new system fully online in short order.  Hopefully the new phone line will also improve speed.

Old Phone Wiring

New Phone Wiring

I survived (sort of).

It has been almost 3 months since my last post. During that time, I have expanded my pool of arm systems and added some significant infrastructure improvements including a 4TB dedicated file server to replace heavily overloaded 1TB server that runs everything. This new server was built for a little more than the cost of a good monitor (~$250 US). The advantages are that instead of two 500Gb drives stripped, I now have 4 1TB drives in a full raid. I can now mirror the entire arm tree, not just main and restricted. This will also include the source, but I will wait a bit to start pulling that until just before release.

I have also learned a great deal about the various server loads and setups. The stuff that wasn't well documented I have written up on my testing wiki. Other tests are already documented within ubuntu.com, either under testcases or general wiki pages. Now that I have these documented and tested, I can turn that documentation into some sort of automation run (well, for most of the tests).

The biggest test I ran was taking all 6 pandas and turning them into a clustered filesystem. Imagine a cluster of cell phones...how cool is that? Slow, but still cool. I also helped track an issue with the USB on these devices where USB drive I/O performance would increase 10x if you flood-pinged the system while it was doing heavy I/O. Still slower than USB on a PC, but at least it is now respectable.

Unfortunately, with all of the server work I have been doing, I have been unable to work on porting Ubuntu to my Nook Color. The current port that I have seen online relies on VNC, with no on-screen support. WTF? Since the main SOC is very close to the same as a BeagleXM or Droid 2, it shouldn't be too hard to figure out. Just need time.

Well, there are 3 weeks left before release. And since we are in Final Freeze, only critical bugs are getting fixed, so that lets me relax on my daily testing a little. I plan to use the time working on automating the server tests, and also writing up blueprints for the next release cycle. Maybe I will get more hardware, like an actual ARM server.

One can dream.