Gentoo md (Software RAID) RAID5 disk crash

May 13th, 2008

It’s spring, so the weather outside is outstanding. The days are longer, the birds are singing and people enjoy the nature wakening up for a new summer of growth.
Having my exams in near future, I’m content that I can’t use a great deal of my time enjoying the spring yet for another few weeks. But at least I’ve set aside the time I need for preparing my exams…
…, I thought. Then, in the middle of my mail inbox I’ve received a mail from Charlie Root at my Gentoo server. Hardly a request for a bicycle ride in the sunshine:
WARNING: Some disks in your RAID arrays seem to have failed! is the message.

Damn, the third year in a row(!) something erroneous happens to my server in the middle of my exam preparations. Why can’t it happen during fall, when the weather’s bad and I have the time for such challenges.
Well, I suppose it’s Murphy’s law. However, over to a more precise description of the problem…

The problem

I have 6 SATA disks in a RAID5 software array (through the use of Gentoo’s md), on which an XFS filesystem is mounted.

The actual error is that two(!) of my disks became faulty overnight. Diagnostics follow.

/proc/mdstat:
md0 : active raid5 sdf1[5] sde1[6](F) sdd1[3] sdc1[2] sdb1[1] sda1[7](F)
(some number) blocks level 5, 64k chunk, algorithm 2 [6/4] [_UUU_U]

I really got nervous here; 2 faulty disks in a RAID5 array means trouble. But why on earth did two disks fail at one time?? I’m not sure yet.

dmesg excerpt:
md: Autodetecting RAID arrays.
md: autorun ...
md: considering sdf1 ...
md: adding sdf1 ...
md: adding sde1 ...
md: adding sdd1 ...
md: adding sdc1 ...
md: adding sdb1 ...
md: adding sda1 ...
md: created md0
md: bind
md: bind
md: bind
md: bind
md: bind
md: bind
md: running:
md: kicking non-fresh sde1 from array!
md: unbind
md: export_rdev(sde1)
md: kicking non-fresh sda1 from array!
md: unbind
md: export_rdev(sda1)
raid5: device sdf1 operational as raid disk 5
raid5: device sdd1 operational as raid disk 3
raid5: device sdc1 operational as raid disk 2
raid5: device sdb1 operational as raid disk 1
raid5: not enough operational devices for md0 (2/6 failed)
RAID5 conf printout:
— rd:6 wd:4
disk 1, o:1, dev:sdb1
disk 2, o:1, dev:sdc1
disk 3, o:1, dev:sdd1
disk 5, o:1, dev:sdf1
raid5: failed to run raid set md0
md: pers->run() failed …
md: do_md_run() returned -5
md: md0 stopped.
md: unbind
md: export_rdev(sdf1)
md: unbind
md: export_rdev(sdd1)
md: unbind
md: export_rdev(sdc1)
md: unbind
md: export_rdev(sdb1)

It seems like sda is the only faulty one (mdadm –examine says it doesn’t have a superblock at all), but the crash of this one must have messed up the superblock of sde. However, it (the superblock of sde) was recreatable, and thereby also the RAID5-array as a whole: RAID5 allows for one disk (in this case, sda) to fail, but then you have no extra parachute, until you replace that faulty one.

The (temporary) solution

The superblock information for the different disks in the RAID-array were extracted for each disk with
mdadm --examine /dev/sda1
and so on. I noted (that is, copy-paste) the exact information for all disks, for (at least) using as option input to the mdadm –create-command in the next paragraph.

I ran (Warning: do this at your own risk! It’s incredibly important setting the right options here, so be sure you’ve read and understand the contents of man mdadm first! Failing to do so will result in loss of data!):
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=6 missing /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1, which put the array back up in degraded mode, with the erroneous disk sda set to missing. This was sufficient to be able to mount the array, but not without problems.

Mounting the degraded array

A first attempt on “mount /dev/md0 /mnt” gave the error “mount: Structure needs cleaning”. This is the XFS filesystem telling it’s not entirely consistent. I could possibly run xfs_repair (I did, in a pretend-type-mode, with the -n option), but I’m not willing to risk my data on this yet. Instead, I did get the device mounted with this command:
mount -r -o norecovery /dev/md0 /mnt

In this way, I can now access my data and make a backup of them. Some data is probably corrupted without repairing the XFS file system, but hopefully most of it is recoverable…

Some of the pages I visited in my frustration

Saving as JPEG with Photoshop

April 21st, 2008

After having edited a photo from camera with 16-bit AdobeRGB color, here is what I do to save as JPEG while maintaining finetuned colors and tones:

  • Flatten image (from many layers to one layer).
  • Convert to 8 bit (JPEG doesn’t support 16 bit): Image -> Mode -> 8 bits/Channel
  • Convert Color Profile (Edit->Convert To Profile…) to sRGB.
    Important to choose Convert to… and not Assign…!
  • Then save as JPEG using either Save As or Save for Web & Devices. Use the first to preserve EXIF tags from camera, use the latter if you want to finetune compression and target file size.

Read more about color spaces/profiles and skin tones at SmugMug.com

Ruby on Rails: Experimenting with ActiveScaffold

April 17th, 2008

Introduction

Here I will set up a sample Ruby on Rails application with the ActiveScaffold scaffolding plugin. I will first give the basic instructions to a default setup (as found on ActiveScaffold’s site, only with my field names and values), and will then make some tweaks and override some of the default configuration. It’s all very basic, but it can be a help for those trying out ActiveScaffold for their first time.

The case we’re working on: create a list of Equipment, i.e. electronic gadgets you’re planning to buy. We want to register title, price, an url to the product page of a webstore, an image url, some description and whether it’s in the webstore’s stock. Aditionally, we would like to calculate the registered price into another currency. All of this is to be shown in a sortable, searchable list.

It will be looking like this when we’re done.

After some years developing and experimenting with PHP, I’ve become curious of Ruby on Rails and what it has to offer in ease and “speedy” development. I’m for the time being an absolute rookie in RoR, but I’ve had a look on some resources on the net and have tried making a few very basic “test” applications.

Having limited experience regarding web development frameworks, it takes a few scratches on the head to get the concept of MVC and knowing what does what (M=model, V=view, C=controller) and what calls what. Generating a scaffold and investigating the different files helps grasping the concept after a while.

The first time I saw the result of a scaffold-generation, I was convinced that this kind of development was something to look further into. However, after having seen samples of other scaffold generators, the default one seems a little static and cumersome to override: It seems like the default scaffolding generator is meant for running once, and then hardcode-editing the scaffold-generated code. Then, if you for some reason have to rescaffold, you must reimplement the changes. Also, the result is rather static without “fancy stuff” like Ajax.

Therefore, I had to test out the ActiveScaffold scaffolding plugin. It’s very easily installed, quickly up and running (a getting-started-tutorial on their site is estimated to 2 minutes from scratch), produces a smooth Ajax’ed design and is very easily to override and configure.

Remember, though, that I’m quite new to RoR, so I haven’t a broad basis of comparison, and there may be better practices for what I’m doing here. If so, please tell me! :-)

Assumptions

I think the majority of the steps in this tutorial should work with most versions (except the scaffolding, which is different in rails v. >= 2), but here’s what I’ve used:

  • Rails 1.2.6 (yes, I know, it’s time to upgrade…)
  • Ruby 1.8.6
  • ActiveScaffold rev. 739

I assume you have a working rails-application, set up with connection to an existing database and accessible through a web browser before starting off.


Basic setup

The first thing we’re gonna do, is create a new table in the database to which the application connects. I created a table called equipment like this in my MySql database:


CREATE TABLE equipment (
  id int(10) unsigned NOT NULL auto_increment,
  title varchar(255) NOT NULL,
  descr text NOT NULL,
  price float NOT NULL,
  updated timestamp NOT NULL
    default CURRENT_TIMESTAMP
    on update CURRENT_TIMESTAMP,
  url varchar(255) NOT NULL,
  imgurl varchar(255) NOT NULL,
  instock tinyint(1) NOT NULL default '1',
  PRIMARY KEY  (id)
) ;

Then, let the generator scripts make the appropriate files we’ll be working on. 4 types of files are created, but we’re only interested in 3 of these; the model, controller and helper files, not the view files. However, we’ll just delete the view files afterwards…
Open up a terminal window, and cd to your rails application root folder. Then run
./script/generate scaffold Equipment

Now, delete the unnecessary views files (not only unnecessary; they also override the ActiveScaffold which we’ll put to work later on, so follow this step!):
rm -R app/views/equipment/

Now it’s about time to install the ActiveScaffold plugin:
([Parts of] the following few steps are borrowed from ActiveScaffold’s website)
./script/plugin install http://activescaffold.googlecode.com/svn/tags/active_scaffold

When the plugin has been downloaded and installed, you need to update a few of your files.
Add this to the head-section of your app/views/layouts/equipment.rhtml file:


  <%= javascript_include_tag :defaults %>
  <%= active_scaffold_includes %>

It should then look something like this:


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
       "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html;charset=UTF-8" />
  <title>as test</title>
  <%= javascript_include_tag :defaults %>
  <%= active_scaffold_includes %>
</head>
<body>

<p style="color: green"><%= flash[:notice] %></p>

<%= yield  %>

</body>
</html>

Then, edit your app/controllers/equipment_controller.rb, so that it looks similar to this:


class EquipmentController < ApplicationController
  @equipment_pages, @equipment = paginate :equipment, :per_page => 10
  layout "equipment"
  active_scaffold :equipment
end

Also, quote from http://activescaffold.com/tutorials/getting-started:

3a. Make sure that you don’t have AjaxScaffold installed in your project. Since ActiveScaffold evolved out of AjaxScaffold they share some common method names and are incompatible with each other.

Then you’re done with basic setup. Now browse to http://your-rails-app-root/Equipment, and hopefully you’ll see the magnificent Ajax’ed layout created by ActiveScaffold. Isn’t it wonderful?

If you, like me, are running Rails on Apache/FastCGI, you may have to restart your Apache httpd-server every now and then (I don’t know of any other methods of killing/restarting the rails fastCGI-scripts being run by apache. If you know; please tell me!). When testing new stuff and/or when getting errors, I always restart apache to see if the problem still persists.

Up to now, we’ve basically dealt with initial setup covered by ActiveScaffold’s own Getting Started-tutorial. Now it’s time to tweak the config to our needs.


Tweaking and overriding default ActiveScaffold config

I’m over all very satisfied with the looks and functionality of the initial output, especially when considering it almost takes no time at all. However, since ActiveScaffold can’t know how I plan to view my data, there are some overrides to be done. Datafields are overridden very smoothly by creating small methods with designated names in the right classes.

Anyways, here’s what I’m gonna do:

  • Show the image pointed to by imgurl instead of the url itself.
  • Put the image inside an <a href…>-tag, so that the image links to the url in the url-field.
  • Create/include a virtual column that shows the price in a foreign currency as well as in USD.
  • Format the two price-columns with sprintf
  • Reorder the columns
  • Exclude the descr-column from list view
  • Tweak the output of the boolean field instock
  • Apply a default sorting of the rows in the list view

This may sound as a lot of work, but fact is it’s quite easily implemented.

First a note on the two price-columns: I live in Norway, and am sometimes interested in monitoring prices in some US webshops. Therefore, I register the USD price in the price-field in the database, while I calculate what this will cost me in NOK (Norwegian kroner) based on the current exchange rate. Since most of you probably are not from Norway, I called NOK the foreign currency.

First, place a floating point number (for instance 4.99 - the actual rate on 2008-Apr-15, the lowest rate in 25 years!) as the only line in a file placed here (though customized for your rails-path!):
/usr/local/www/rails/as/myincludes/nok_usd_rate.txt

We want a global variable (yeah, I know everyone’s not found of that) to hold the exchange rate, read from a file in the event of page loading. I do this so that I won’t have to read the file over for each record.

Edit the app/controllers/application.rb-file, so that it looks like this:


# Filters added to this controller apply to all controllers in the application.
# Likewise, all the methods added will be available for all controllers.

class ApplicationController < ActionController::Base
  def setmyglobalvar
    # reads exchange rate from file: UPDATE TO YOUR PATH!
    contents = File.read('/usr/local/www/rails/as/myincludes/nok_usd_rate.txt')
    # converts to NOK and includes 25% "mva" (=Norwegian VAT). Stores in global var:
    $myglobalvar= contents.to_f * 1.25
  end

  # Pick a unique cookie name to distinguish our session data from others'
  session :session_key => '_testapp_session_id'

  before_filter :setmyglobalvar
end

Here, we have defined a method, and calls this by the before_filter-statement.

Next, we must extend our record-model so that it includes the field to be used as a virtual column. Edit app/models/equipment.rb to look like this:


class Equipment < ActiveRecord::Base
  def price_nok
    # $myglobalvar is set in controllers/application_controller,
    # and contains exchange rate incl VAT:
    # self.price references the price-
    # field from current database record.
    self.price * $myglobalvar
  end
end

Then we must tell the controller to include this new field as a (virtual) column.
At the same time, we reorder the columns and apply a default sorting for the list view.
Note that we only alter the columns of the list action and not of any other actions (list is an action, so is new, edit, show etc). Altering the columns of edit or new can result in an error on record edit or creation attempts.
Note also that only the columns listed in config.list.columns will be shown in the list view.
Here it goes - app/controllers/equipment_controller.rb:


class EquipmentController < ApplicationController
  @equipment_pages, @equipment = paginate :equipment, :per_page => 10
  layout "equipment"

  active_scaffold :equipment do |config|
    list.sorting = {:price => 'DESC'}
    config.list.columns = [:imgurl, :instock, :price, :price_nok, :title, :updated]
  end
end

Then, at last, we alter the way some of the columns are shown. This is done in app/helpers/equipment_helper.rb, by creating methods with name columnname_column. All output within the method will be printed in the respective table cell (td) for each record of that column. Here is the contents of that file:


module EquipmentHelper
  def imgurl_column(record)
    '<a href="'+record.url+'">' + image_tag(record.imgurl, :alt => "Image") + '</a>'
  end
  def url_column(record)
    '<b><a href="'+record.url+'">[ link ]</a></b>'
  end
  def instock_column(record)
    if record.instock
      image_tag("/img/yes.png", :alt => "yes!")
    else
      image_tag("/img/no.png", :alt => "no, sadly!")
    end
  end
  def price_column(record)
    # http://railsmanual.org/module/ActionView::Helpers::NumberHelper
    number_to_currency(record.price)
  end
  def price_nok_column(record)
    number_to_currency(record.price_nok, :unit => "NOK ", :delimiter => " ", :separator => ",")
  end
  def descr_column(record)
    # for the show action, where we want to show the description as well.
    record.descr.gsub("\n", "<br />")
  end
end
Custom stylesheet

If you want to apply your own css-formatting, don’t edit the default stylesheets, because they may be overwritten. Instead, include this as the very last line in the head-section of app/views/layouts/equipment.rhtml:


  <%= stylesheet_link_tag 'active_scaffold_overrides' %>

Then, add your css rules in the stylesheet public/stylesheets/active_scaffold_overrides.css! For instance:


.price_nok-column
{
        font-weight: bold;
}


Links

The last days of my Logitech MX3200 wireless keyboard/mouse?

April 7th, 2008

I found this very thorough review regarding the MX5500, with some comparisons to the MX3200:

http://aphnetworks.com/reviews/logitech_mx5500_revolution

I’ve had my MX3200 for a year and five months now, and it has been alright, but the last few months I’ve experienced times of frustration:
Sometimes when I write a character (which requires the shift-button??), the character is being repeated until I hit another key, even though I released the first key. The problem is not related to the key itself, so I guess it’s a problem regarding weak wireless reception or something. When the problem first occurs, it can happen over and over again for hours, while other times there can be days/weeks when there’s no problem at all. This makes me suspect that maybe the signal/noise ratio is too low; maybe the signal reception is too weak on this model and that the problem occurs when the noise from cellulars etc is having its peaks?

What made me suspect the reception quality in the first place, was when I first plugged the receiver in the back of my computer, curious to try out my new keyboard. The reception was less than low: Not all keystrokes were recorded, and I had to use the equipped USB extension cable to place the receiver nearer my keyboard. Considering that the back of my computer is no more than 1,5 m away from the keyboard — though visually hidden by the cabinet itself — I think that’s poor reception.

Anyways, I haven’t had this problem with any other (Logitech) wireless keyboards I’ve had, so I hope and think it’s just this model. Therefore, I found the review linked to above interesting, especially since they mention good reception on the MX5500.

Escaping backslash character in C# .NET MySQL-queries

March 12th, 2008

Using escape-sequences like \t or \n is one thing, but escaping the escapecharacter (often ‘\’) can sometimes be a pain in the A*S*S… This is especially true when the string you’re escaping is about to be processed by several succeeding processes, like: 1) a compiler, 2) a regex-engine, 3) a database, … , etc.

Anyways, I’ve experimented some with MySql-queries from within C# .NET, using Windows paths (containing backslashes as path separators) in the WHERE-clauses. We have a query of one of the forms:

case 1: = operator

querystring1 =
  "SELECT a,b,path FROM table " +
  "WHERE path = '" + patharg1 + "';";

or

case 2: LIKE operator

querystring2 =
  "SELECT a,b,path FROM table " +
  "WHERE path LIKE '" + patharg2 + "';";

Here is what I’ve found out:

case 1: = operator
Use 4 backslashes for each backslash in the saved record you’re matching against. You could do:

patharg1 =
  patharg1.Replace("\\", "\\\\");

(The 2 backslashes in the first replace-argument is for escaping the backslash within the compiler.)

case 1: LIKE operator
Use 8(!) backslashes for each backslash in the saved record you’re matching against. You could do:

patharg2 =
  patharg2.Replace("\\", "\\\\\\\\");

(The 2 backslashes in the first replace-argument is for escaping the backslash within the compiler.)

The two cases could then look like this:

case 1:

querystring1 =
  "SELECT a,b,path FROM table " +
  "WHERE path = 'c:\\\\some\\\\path.txt';";

case 2:

querystring2 =
  "SELECT a,b,path FROM table " +
  "WHERE path LIKE 'c:\\\\\\\\some\\\\\\\\path.txt';";

The above is if you put the querystring into, for instance, a MySqlCommand’s CommandText.

However, if you use datasets and run something like

dataTable.Select(filterExpression)

where dataTable is a DataTable instance and filterExpression is a string, you must not escape the backslash more than for a usual string (you’ll just write “\\” or @”\”, because some kind of escape of the backslash is always necessary for the compiler).

Note to self: C# and logical shortcircuiting

March 10th, 2008

C# (.NET) do short-circuit logical expressions; i.e.:

1)

(A || B)

==> If A is true, then only A is evaluated.

2)

(C && D)

==> If C is false, then only C is evaluated.

This means that, provided that A is true, expression 1 is true even though B would throw an exception if evaluated.

An example use of expression 2:


if ((filename != "") && (new FileInfo(filename).Exists))
	//file exists

If filename == “”, the Exists-statement would throw an exception. However, since the first expression (…!=…) is false, the Exists-statement is never evaluated, and the exception doesn’t occur.

New Garmin Colorado 300 vs old Garmin 60CS : First impressions review

February 29th, 2008

Dear readers!

I’ve recently bought myself a new GPS, namely the new Garmin Colorado 300, replacing my old Garmin GpsMap 60CS.
Read the rest of this entry »

Welcome!

February 29th, 2008

Welcome to AkillesBlog!

This is my blog’s very first post, and there will be a new one every now and then.

I’m not yet (and probably never will be) ready to define an agenda for this blog, but I think it will be used to memorize for myself - as well as sharing with others who may be interested - thoughts, techiques, tests, “reviews” and so on about general stuff.

Being somewhat of a geek taking great interest in computer programming, photography and electronic widgets, I think you’ll understand my meaning of general stuff;-)

Anyway, please understand that this site is purely subject to my personal, subjective thoughts, and that I do NOT take any responsibility whatsoever for what anyone might think or do as a result of this site. I do not guarantee the correctness or accuracy of any of the contents either.

Happy surfing!

Best regards,
Christian