Friday, December 17, 2010

Shell escaped file names in ruby

When files have crazy characters in their filename (like ' or ""), it can make creating a robust script challenging. After all, if you write the script with either a single quotation mark or a double quotation mark, the other one will trip you up:


file1 = "my'crazy'file.txt"
file2 = 'my"crazy"file.txt'
system "mv \"#{file2}\" bettername.txt"  # fails on file2
system "mv '#{file2}' bettername.txt"    # fails on file1

The answer is to use the multiple arguments invocation, which will properly escape your filename to be fully shell compatible:

system "mv", file1, "bettername.txt"  # works on file1 or file2

Thursday, December 9, 2010

Playing with (compiling) the redesigned NArray

Masahiro Tanaka has been redesigning NArray. Word is that it may be pulled into mainline ruby.

So, what is it like? Amazing from what I can tell. How do you play with it? At least on Ubuntu 10.10, this is what I did (there may be other dependencies I've neglected here):


sudo apt-get install libatlas-dev libatlas-base-dev
# modify extconf.rb:
have_header("atlas/cblas.h")   ===>   have_header("cblas.h")
# modify linalg.c
#include <atlas/cblas.h>   ===>   #include <cblas.h>
# for ruby 1.9, you also need to change a few normal array things:

diff -r ../narray-new-1.8/narray.c ./narray.c
211,212c211,212
<     n = RARRAY(idxargs)->len;
<     ptr = RARRAY(idxargs)->ptr;
---
>     n = RARRAY_LEN(idxargs);
>     ptr = RARRAY_PTR(idxargs);
232c232,233
< const static size_t zero=0;
---
> //const static size_t zero=0;
> static const size_t zero=0;
463c464
<      ndim = RARRAY(v)->len;
---
>      ndim = RARRAY_LEN(v);
761a763
>     VALUE *ptr;
774,775c776,778
<  RARRAY(v)->ptr[i] = SIZE2NUM(na->shape[c]);
<  RARRAY(v)->len++;
---
>  RARRAY_PTR(v)[i] = SIZE2NUM(na->shape[c]);
>  ptr = RARRAY_LEN(v);
>     ptr++;
diff -r ../narray-new-1.8/nstruct.c ./nstruct.c
301c301
<      ndim = RARRAY(argv[i])->len;
---
>      ndim = RARRAY_LEN(argv[i]);

now, you should be able to "ruby extconf.rb", "make", and then require 'narray' or require './narray.so'

Tuesday, September 28, 2010

Compile ruby 1.9.2 on Ubuntu

assumes you have build-essential package installed
wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p0.tar.bz2
tar -xjvf ruby-1.9.2-p0.tar.bz2
cd ruby-1.9.2-p0
./configure --enable-shared --program-suffix=1.9.2
make
make check # optional
sudo checkinstall make install
# [follow the defaults until here:
# NOTE that you want the extra files inside your directory.
# those extra files contain things like the openssl libraries!!

    Some of the files created by the installation are inside the build
    directory: /home/jtprince/src/ruby-1.9.2-p0

    You probably don't want them to be included in the package,
    especially if they are inside your home directory.
    Do you want me to list them?  [n]: n
    Should I exclude them from the package? (Saying yes is a good idea)  [y]: n

# At this point, you can install the package on machines of the same
# architecture with "sudo dpkg -i ruby-1.9.2_p0-1_amd64.deb" [depending on your arch]

sudo rm /usr/bin/ruby
sudo ln -s /usr/local/bin/ruby1.9.2 /usr/bin/ruby

Okay, now you need to get rubygems working properly. Remember, rubygems comes with ruby 1.9.X, so you don't need to download it separately.

At this point, we have to ensure our soft links point where we want and to tell gems to point to ruby 1.9.2

sudo cp /usr/bin/gem1.9.{1,2}
# then edit /usr/bin/gem1.9.2
# change #!/usr/bin/ruby1.9.1 ==> #!/usr/local/bin/ruby1.9.2

# now set up the soft link so that "gem" calls our gem1.9.2
sudo rm /etc/alternatives/gem
sudo ln -s /usr/bin/gem1.9.2 /etc/alternatives/gem

Maybe it is unnecessary, but I moved my local gem folder so that all my gems are compiled from scratch again to ensure they have been built against 1.9.2. how to set up a local gem environment

mv ~/.gem/ruby/{,not_using_}1.9.1
mkdir ~/.gem/ruby/1.9.2

Tuesday, September 21, 2010

Installing graphviz on Ubuntu Lucid Lynx 10.04

If you want sfdp (a great layout for large graphs), then you need the .debs from the graphviz site (not through the ubuntu repos).

Download the develpment snapshots here
note, if you don't use the language specific bindings listed, you don't need to download those.

sudo aptitude install libdevil1c2 libgtkglext1
sudo dpkg -i graphviz_*.deb graphviz-dev*_all.deb 
graphviz-doc*_all.deb libgraphviz4_*.deb libgraphviz-dev_*.deb libgv-ruby_*.deb

Now, you can run sfdp!

Sunday, September 5, 2010

for anything complicated, ruby eats bash for breakfast

http://www.sklav.com/node/4 shows a bash solution to the issue debated here. Regardless of how good a bash script is, it inevitably is not very DRY, I think because it is hard to be DRY in bash scripting.

Here is a ruby solution. Of course, all I did was factor, which is easy in ruby:


#!/usr/bin/env ruby

lame_opts = "--preset standard -q0"

if ARGV.size == 0
  puts "usage: #{File.basename(__FILE__)} .flac ..."
  puts "output: .mp3"
  puts ""
  puts "uses lame opts: #{lame_opts}"
  puts "retains tag info"
  exit
end

tag_convert = {
  :tt => "TITLE",
  :tl => "ALBUM",
  :ta => "ARTIST",
  :tn => "TRACKNUMBER",
  :tg => "GENRE",
  :ty => "DATE",
}

ARGV.each do |file|
  tag_opts = tag_convert.map do |key,val|
    # comes out as TITLE=
    data = `metaflac --show-tag=#{val} #{file}`.split("=",2).last
    "--#{key} #{data}"
  end
  mp3name = file.chomp(File.extname(file)) + ".mp3"
  `flac -dc #{file} | lame #{lame_opts} #{tag_opts.join(" ")} --add-id3v2 - #{mp3name}`
end
Fairly easy to follow, especially since we don't repeat ourselves. Most of the code is in creating a useful help message.

For Perl hackers, here is the entire script on a few lines of less than 105 chars per line:

tag_convert = {:tt=>"TITLE",:tl=>"ALBUM",:ta=>"ARTIST",:tn=>"TRACKNUMBER",:tg=>"GENRE",:ty=>"DATE"}
ARGV.each do |f| 
  tgs = tag_convert.map {|k,v| "--#{k} "+`metaflac --show-tag=#{v} #{f}`.split("=",2).last}.join(" ")
  `flac -dc #{f} | lame --preset standard -q0 #{tgs} --add-id3v2 - #{f.sub(/.flac$/i,".mp3")}`
end
There are still ways to compress this further, but it is very concise and surprisingly easy to follow (if you know ruby).

Wednesday, September 1, 2010

For sets of integers, lookup speed goes: Array > Hash > Set

You have some set of integers and you need to query it repeatedly to determine if a member is in it or not.

require 'benchmark'
require 'set'

ar = (0...1000).to_a

ar_index = []
hash_index = {}
ar.each do |v|
  ar_index[v] = true
  hash_index[v] = true
end

set_index = Set.new(ar)
access = (0...1000).to_a

TIMES = 10000

Benchmark.bmbm do |r|
  r.report("array index") { TIMES.times { access.each {|v| ar_index[v] } } }
  r.report("hash index") { TIMES.times { access.each {|v| hash_index[v] } } }
  r.report("hash include") { TIMES.times { access.each {|v| hash_index.include?(v) } } }
  r.report("set include") { TIMES.times { access.each {|v| set_index.include?(v) } } }
end
The output (ruby 1.9.1p378 (2010-01-10 revision 26273) [x86_64-linux]):

                   user     system      total        real
array index    0.730000   0.000000   0.730000 (  0.745341)
hash index     0.990000   0.000000   0.990000 (  1.003517)
hash include   1.310000   0.000000   1.310000 (  1.332846)
set include    1.970000   0.000000   1.970000 (  2.024808)

It's faster to numerically round than to sprintf in ruby

require 'benchmark'

class Float
  def round_to(x)
    (self * 10**x).round.to_f / 10**x
  end

  def ceil_to(x)
    (self * 10**x).ceil.to_f / 10**x
  end

  def floor_to(x)
    (self * 10**x).floor.to_f / 10**x
  end

  def round_to_sprintf_f(x)
    ("%.#{x}f" % x).to_f
  end

  def round_to_sprintf_sprintf(x)
    sprintf("%.#{x}f", x)
  end

  def round_to_sprintf_modulus(x)
    "%.#{x}f" % x
  end

end

num = 34.2342134
place = 3

TIMES = 1000000
Benchmark.bmbm do |r|
  r.report("numeric round_to (float)") { TIMES.times { num.round_to(place) } }
  r.report("numeric round_to (to string)") { TIMES.times { num.round_to(place).to_s } }
  r.report("sprintf (float)") { TIMES.times { num.round_to_sprintf_f(place) } }
  r.report("sprintf (string)") { TIMES.times { num.round_to_sprintf_sprintf(place) } }
  r.report("sprintf modulus (string)") { TIMES.times { num.round_to_sprintf_modulus(place) } }
end
The output:

% ruby -v
ruby 1.9.1p378 (2010-01-10 revision 26273) [x86_64-linux]
...
                                   user     system      total        real
numeric round_to (float)       0.820000   0.000000   0.820000 (  0.819920)
numeric round_to (to string)   1.990000   0.000000   1.990000 (  1.989912)
sprintf (float)                3.620000   0.000000   3.620000 (  3.614758)
sprintf (string)               2.650000   0.000000   2.650000 (  2.654812)
sprintf modulus (string)       3.190000   0.000000   3.190000 (  3.191807)

The conclusion: Regardless of whether you are going to a string or numeric after rounding, it is faster to round numerically than with sprintf. Also, using the sprintf function is faster than '%'. Usually speed doesn't matter, but when it does...

Thursday, August 19, 2010

apply & sweep : Column-wise & Row-wise operations in R

How do you do operations on rows and columns in R? apply and sweep are the main tools. Here is an example of "autoscaling" (mean center and divide by standard deviation) on each column:

[I verified that this is correct on oocalc]

# 1 -> across the rows
# 2 -> across the columns 
m <- matrix(data=c(1,2,3,4,4.5,6,7,8,9,10,11,13), nrow=4, ncol=3)
m
#      [,1] [,2] [,3]
# [1,]    1  4.5    9
# [2,]    2  6.0   10
# [3,]    3  7.0   11
# [4,]    4  8.0   13

colmeans <- apply(mymatrix, 2, mean)  # the column-wise means
# mc = the column-wise mean centered data
mc <- sweep(m, 2, colmeans, "-")   # subtract is the default

col_stdev <- apply(m, 2, sd)  # column-wise standard deviations

mcstd <- sweep(mc, 2, col_stdev, "/")  # divide by standard deviations

mcstd
#            [,1]       [,2]      [,3]
# [1,] -1.1618950 -1.2558275 -1.024695
# [2,] -0.3872983 -0.2511655 -0.439155
# [3,]  0.3872983  0.4186092  0.146385
# [4,]  1.1618950  1.0883839  1.317465

Thursday, August 5, 2010

Regression line, R^2 (Pearson's correlation coefficient), slope, and y intercept in R

A basic feature in Excel and OpenOffice Calc is to find and plot the regression line. Here is an example of doing this in R. This is more work in R, but you are sacrificing for more power.

Here is how we get the data out:

x = c(1,2,3,4,5)
y = c(3,4,5,5.1,6.2)
pearsons_r = cor(x,y)
r_squared = pearsons_r^2
fit = lm(y~x) # notice the order of variables
y_intercept = as.numeric(fit$coeff[1])
slope = as.numeric(fit$coeff[2])

Here is how we plot it:

plot(x,y)
abline(lm(y~x)) # again, notice the order
function_string = paste("f(x) = ", slope, "x + ", y_intercept,  sep="")
r_sq_string = paste("R^2 =", r_squared)
display_string = paste(function_string, r_sq_string, sep="\n")
mtext(display_string, side=3, adj=1)  # top right outside of the margin

Thursday, June 17, 2010

clone a VDI Virtual Box image for using elsewhere

How robust is this procedure (found here)?
  1. Shut down the virtual machine you would like to copy
  2. In File > Virtualdiskmanager, select the virtual machine disk image you would like to copy, and press the Release button
  3. In a terminal window, issue following command (see virtualbox user manual):
    vboxmanage clonevdi /directory/image1.vdi /directory/image2.vdi
  4. In File > Virtualdiskmanager, add the new disk image you've created in step 3
  5. In the main virtualbox window, press the New button to create a new virtual machine, and link it to the new disk image you've created.

Wednesday, June 9, 2010

Cartoons in biology?? How about: diagram, figure, illustration or schematic

Molecular biologists often refer to depictions of cellular processes or structure as cartoons.  (here is an example).  I think it is a poor word choice.  Cartoon is generally used to indicate a humorous depiction or an unfinished product (i.e., a sketch on a carton), so it is somewhat insulting to the creator of the illustration (unless it is, in fact, unfinished or humorous).  What I think they are trying to say by calling their diagram a cartoon is something like: "do not forget that this diagram is a vast oversimplification of reality!  Proteins are not really boxes and lines are really not protein associations!  Proteins aren't made of ribbons!"  At times, one also gets the feeling they are trying to convey something more, "Our field is so complex, our diagrams are ludicrously simple compared to reality!" or even, "Aren't I cute for calling this a cartoon even though this is a science talk!"

Fortunately, there are several other words suggesting that a depiction is not a precise representation of reality but rather a simplification that serves to emphasize certain facts or relationships, and they don't carry any of cartoon's demeaning connotations: diagram, figure, illustration, schematic.  IMHO, these are vastly preferable for referring to depictions of biological phenomenon.  This seems to be well supported by definitions, synonyms, and word associations:


Merriam-Webster (online)

diagram
Etymology: Greek diagramma, from diagraphein to mark out by lines, from dia- + graphein to write — more at carve
Date: 1619
1 : a graphic design that explains rather than represents; especially : a drawing that shows arrangement and relations (as of parts)
2 : a line drawing made for mathematical or scientific purposes

schema
Inflected Form(s): plural sche·ma·ta \-mə-tə\ also schemas
Etymology: Greek schēmat-, schēma
Date: circa 1890
1 : a diagrammatic presentation; broadly : a structured framework or plan : outline
 
figure
Etymology: Middle English, from Anglo-French, from Latin figura, from fingere
Date: 13th century
2 a : a geometric form (as a line, triangle, or sphere) especially when considered as a set of geometric elements (as points) in space of a given number of dimensions b : bodily shape or form especially of a person c : an object noticeable only as a shape or form
s moving in the dusk>

3 a : the graphic representation of a form especially of a person or geometric entity b : a diagram or pictorial illustration of textual matter

illustration

Date: 14th century
2 : something that serves to illustrate: as a : an example or instance that helps make something clear b : a picture or diagram that helps make something clear or attractive

cartoon
Etymology: Italian cartone pasteboard, cartoon, augmentative of carta leaf of paper — more at card
Date: 1671
1 : a preparatory design, drawing, or painting (as for a fresco)
2 a : a drawing intended as satire, caricature, or humor b : comic strip
3 : animated cartoon
4 : a ludicrously simplistic, unrealistic, or one-dimensional portrayal or version ;

Thesaurus.com

diagram

Part  of  Speech:      noun
Definition:     drawing, sketch of form or plan
Synonyms:     big picture, blueprint, chart, description, design, draft, figure, floor plan, game, game plan, ground plan, layout, outline, perspective, representation, rough draft

schema
Part  of  Speech:      noun
Definition:     design
Synonyms:     architecture, arrangement, blueprint, chart, comp, composition, conception, constitution, construction, delineation, depiction, diagram, doodle, drawing, dummy, form, formation, game plan, idea, layout, makeup, map, method, model, outline, paste-up, pattern, perspective, picture, plan, project, scheme, strategy, study, tracery, tracing, treatment

figure
Part  of  Speech:      noun
Definition:     object with design; depiction
Synonyms:     cast, composition, decoration, device, diagram, drawing, effigy, embellishment, emblem, illustration, image, model, mold, motif, motive, ornamentation, pattern, piece, portrait, representation, sketch, statue

illustration
Part  of  Speech:      noun
Definition:     drawing, artwork that assists explanation
Synonyms:     adornment, cartoon, decoration, depiction, design, engraving, etching, figure, frontispiece, halftone, image, line drawing, painting, photo, photograph, picture, plate, sketch, snapshot, tailpiece, vignette

cartoon
Part  of  Speech:      noun
Definition:     funny drawing, often with dialogue or caption
Synonyms:     animation, caricature, comic strip, drawing, lampoon, parody, representation, satire, sketch, takeoff


www.wordassociation.org


associated to diagram
1.    chart    weak
2.    graph    weak
3.    draw    v. weak
4.    plan    v. weak
5.    block    v. weak
   
associated from diagram
1.    chart    medium
2.    picture    medium
3.    diaphragm    weak
4.    drawing    v. weak
5.    graph    v. weak

associated to schema
1.    scheme    strong
2.    axiom    weak
3.    concept    weak
4.    fema    weak
5.    pooh    weak
   
associated from schema
1.    scheme    strong
2.    database    weak
3.    diagram    weak
4.    paradigm    weak
5.    seem    weak

associated to figure
1.    stick    medium
2.    action    weak
3.    father    weak
4.    shape    weak
5.    authority    v. weak
   
associated from figure
1.    shape    medium
2.    eight    medium
3.    skate    weak
4.    skating    v. weak
5.    out    v. weak

associated to schematic
1.    plan    strong
2.    scheme    medium
3.    architecture    weak
4.    blueprint    weak
5.    diagram    weak
   
associated from schematic
1.    blueprint    strong
2.    system    strong
3.    design    medium
4.    drawing    medium
5.    love    medium

associated to illustration
1.    example    medium
2.    drawing    medium
3.    graphic    weak
4.    book    v. weak
5.    capella    v. weak
   
associated from illustration
1.    picture    strong
2.    drawing    medium
3.    book    weak
4.    cartoon    weak
5.    draw    weak

associated to cartoon
1.    animation    v. weak
2.    simpsons    v. weak
3.    anime    v. weak
4.    disney    v. weak
5.    character    v. weak
   
associated from cartoon
1.    character    medium
2.    network    weak
3.    animation    weak
4.    comic    v. weak
5.    tv    v. weak