ruby list comprehension

I have seen list comprehension cited as a feature of python that ruby lacks. I want to play around with the idea to see where it goes...

[disclaimer: this is just scratch space for ideas at the moment]

(I wonder how lazy enumerators will change the ruby approach?)

A nice write up on list comprehensions in python has an example where you can use list comprehension to find prime numbers. They divide it into two steps because it gets a little hairy on one line. Here's finding primes on one line using list comprehension:

>>> [x for x in range(2, 50) if x not in [j for i in range(2, 8) for j in range(i*2, 50, i)]]
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]

It's not bad, but it's easy to get lost in there (my ruby eyes are not highly trained in python, of course).

Now, one of the things list comprehensions do is it does selection and modification in the same phrase. This part of list comprehensions is easy to implement.

[ add the equivalent python here ]

(2..10).select(&:even?).map {|v| v**2 }
(2..10).map {|v| v**2 if v.even? }.compact

Implementing this aspect of list comprehensions in ruby is less than trivial:

module Enumerable
  def lc(&block)
    block.nil? ? self : self.map(&block).compact
  end
end

The other part of list comprehensions is the fact that the evaluation happens from right to left (in the python code) so that the execution of the code is efficient.

This gets the job done, but the right-most block is executed multiple times to make it work:

(2..50).reject {|j| (2..8).map {|i| (i*2).step(50,i).to_a}.flatten.include?(j) 

# also other ways to think about this (need to finish this post)
Set.new(2..50) - (2..8).reduce(Set.new) {|np,i| np.merge((i*2).step(50,i)) }

Using Dropbox and git

Dropbox is really fantastic, but its version control abilities are limited. Git is a fantastic tool for versioning things.  Here's how I use both to keep detailed version history of the projects that matter to me:

Solo Repo (use a Normal git repo)

For most projects, I do not need to merge changes with multiple people, but I want a version history. For these, I set up a normal git repository. This repo has no "origin", so I never push to anything, just stage and checkin.

cd ~/Dropbox/myproject
git init
git add .
git commit -m 'first commit'
# after a while
git status
git commit -a -m 'another commit' 

Easy to do, and all my changes (even when I don't happen to check them in) follow me on all the computers I use dropbox on.

Shared Repo (use a Bare git repo)

In this case, I'll make a bare git repo and then share the folder with others. I'll assume I'm starting with an existing project directory, but it is not a git repo yet. (there are other ways to do this).

# my project is sitting on my local machine somewhere.
# I will make it a normal git repository
cd someproject
git init
git add .
git commit -m 'first commit'
# now make a bare git repository on my dopbox folder
# by convention, bare repos end in '.git'
mkdir -p ~/Dropbox/git/someproject.git  
pushd ~/Dropbox/git/someproject.git 
git init --bare
popd  # back where I was
git remote add origin ~/Dropbox/git/someproject.git
# now we can push and pull to our origin (see below if you get errors)
git pull
git push

If you get trouble when pushing or pulling to your remote, add these lines to your ~/.gitconfig file:

[push]
 default = matching
[branch "master"]
 remote = origin
 merge = refs/heads/master

Shell escaped file names in ruby

When files have crazy characters in their filename (like ' or ""), it can make creating a robust script challenging. After all, if you write the script with either a single quotation mark or a double quotation mark, the other one will trip you up:

file1 = "my'crazy'file.txt"
file2 = 'my"crazy"file.txt'
system "mv \"#{file2}\" bettername.txt"  # fails on file2
system "mv '#{file2}' bettername.txt"    # fails on file1

The answer is to use the multiple arguments invocation, which will properly escape your filename to be fully shell compatible:

system "mv", file1, "bettername.txt"  # works on file1 or file2

Playing with (compiling) the redesigned NArray

Masahiro Tanaka has been redesigning NArray. Word is that it may be pulled into mainline ruby.

So, what is it like? Amazing from what I can tell. How do you play with it? At least on Ubuntu 10.10, this is what I did (there may be other dependencies I've neglected here):

sudo apt-get install libatlas-dev libatlas-base-dev
# modify extconf.rb:
have_header("atlas/cblas.h")   ===>   have_header("cblas.h")
# modify linalg.c
#include <atlas/cblas.h>   ===>   #include <cblas.h>
# for ruby 1.9, you also need to change a few normal array things:

diff -r ../narray-new-1.8/narray.c ./narray.c
211,212c211,212
<     n = RARRAY(idxargs)->len;
<     ptr = RARRAY(idxargs)->ptr;
---
>     n = RARRAY_LEN(idxargs);
>     ptr = RARRAY_PTR(idxargs);
232c232,233
< const static size_t zero=0;
---
> //const static size_t zero=0;
> static const size_t zero=0;
463c464
<      ndim = RARRAY(v)->len;
---
>      ndim = RARRAY_LEN(v);
761a763
>     VALUE *ptr;
774,775c776,778
<  RARRAY(v)->ptr[i] = SIZE2NUM(na->shape[c]);
<  RARRAY(v)->len++;
---
>  RARRAY_PTR(v)[i] = SIZE2NUM(na->shape[c]);
>  ptr = RARRAY_LEN(v);
>     ptr++;
diff -r ../narray-new-1.8/nstruct.c ./nstruct.c
301c301
<      ndim = RARRAY(argv[i])->len;
---
>      ndim = RARRAY_LEN(argv[i]);

now, you should be able to "ruby extconf.rb", "make", and then require 'narray' or require './narray.so'

Compile ruby 1.9.2 on Ubuntu

assumes you have build-essential package installed

wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p0.tar.bz2
tar -xjvf ruby-1.9.2-p0.tar.bz2
cd ruby-1.9.2-p0
./configure --enable-shared --program-suffix=1.9.2
make
make check # optional
sudo checkinstall make install
# [follow the defaults until here:
# NOTE that you want the extra files inside your directory.
# those extra files contain things like the openssl libraries!!

    Some of the files created by the installation are inside the build
    directory: /home/jtprince/src/ruby-1.9.2-p0

    You probably don't want them to be included in the package,
    especially if they are inside your home directory.
    Do you want me to list them?  [n]: n
    Should I exclude them from the package? (Saying yes is a good idea)  [y]: n

# At this point, you can install the package on machines of the same
# architecture with "sudo dpkg -i ruby-1.9.2_p0-1_amd64.deb" [depending on your arch]

sudo rm /usr/bin/ruby
sudo ln -s /usr/local/bin/ruby1.9.2 /usr/bin/ruby

Okay, now you need to get rubygems working properly. Remember, rubygems comes with ruby 1.9.X, so you don't need to download it separately.

At this point, we have to ensure our soft links point where we want and to tell gems to point to ruby 1.9.2

sudo cp /usr/bin/gem1.9.{1,2}
# then edit /usr/bin/gem1.9.2
# change #!/usr/bin/ruby1.9.1 ==> #!/usr/local/bin/ruby1.9.2

# now set up the soft link so that "gem" calls our gem1.9.2
sudo rm /etc/alternatives/gem
sudo ln -s /usr/bin/gem1.9.2 /etc/alternatives/gem

Maybe it is unnecessary, but I moved my local gem folder so that all my gems are compiled from scratch again to ensure they have been built against 1.9.2. how to set up a local gem environment

mv ~/.gem/ruby/{,not_using_}1.9.1
mkdir ~/.gem/ruby/1.9.2

Installing graphviz on Ubuntu Lucid Lynx 10.04

If you want sfdp (a great layout for large graphs), then you need the .debs from the graphviz site (not through the ubuntu repos).

Download the develpment snapshots here
note, if you don't use the language specific bindings listed, you don't need to download those.

sudo aptitude install libdevil1c2 libgtkglext1
sudo dpkg -i graphviz_*.deb graphviz-dev*_all.deb 
graphviz-doc*_all.deb libgraphviz4_*.deb libgraphviz-dev_*.deb libgv-ruby_*.deb

Now, you can run sfdp!

Biostatistics vs Lab Research

for anything complicated, ruby eats bash for breakfast

http://www.sklav.com/node/4 shows a bash solution to the issue debated here. Regardless of how good a bash script is, it inevitably is not very DRY, I think because it is hard to be DRY in bash scripting.

Here is a ruby solution. Of course, all I did was factor, which is easy in ruby:

#!/usr/bin/env ruby

lame_opts = "--preset standard -q0"

if ARGV.size == 0
  puts "usage: #{File.basename(__FILE__)} .flac ..."
  puts "output: .mp3"
  puts ""
  puts "uses lame opts: #{lame_opts}"
  puts "retains tag info"
  exit
end

tag_convert = {
  :tt => "TITLE",
  :tl => "ALBUM",
  :ta => "ARTIST",
  :tn => "TRACKNUMBER",
  :tg => "GENRE",
  :ty => "DATE",
}

ARGV.each do |file|
  tag_opts = tag_convert.map do |key,val|
    # comes out as TITLE=
    data = `metaflac --show-tag=#{val} #{file}`.split("=",2).last
    "--#{key} #{data}"
  end
  mp3name = file.chomp(File.extname(file)) + ".mp3"
  `flac -dc #{file} | lame #{lame_opts} #{tag_opts.join(" ")} --add-id3v2 - #{mp3name}`
end

Fairly easy to follow, especially since we don't repeat ourselves. Most of the code is in creating a useful help message.

For Perl hackers, here is the entire script on a few lines of less than 105 chars per line:

tag_convert = {:tt=>"TITLE",:tl=>"ALBUM",:ta=>"ARTIST",:tn=>"TRACKNUMBER",:tg=>"GENRE",:ty=>"DATE"}
ARGV.each do |f| 
  tgs = tag_convert.map {|k,v| "--#{k} "+`metaflac --show-tag=#{v} #{f}`.split("=",2).last}.join(" ")
  `flac -dc #{f} | lame --preset standard -q0 #{tgs} --add-id3v2 - #{f.sub(/.flac$/i,".mp3")}`
end

There are still ways to compress this further, but it is very concise and surprisingly easy to follow (if you know ruby).

For sets of integers, lookup speed goes: Array > Hash > Set

You have some set of integers and you need to query it repeatedly to determine if a member is in it or not.

require 'benchmark'
require 'set'

ar = (0...1000).to_a

ar_index = []
hash_index = {}
ar.each do |v|
  ar_index[v] = true
  hash_index[v] = true
end

set_index = Set.new(ar)
access = (0...1000).to_a

TIMES = 10000

Benchmark.bmbm do |r|
  r.report("array index") { TIMES.times { access.each {|v| ar_index[v] } } }
  r.report("hash index") { TIMES.times { access.each {|v| hash_index[v] } } }
  r.report("hash include") { TIMES.times { access.each {|v| hash_index.include?(v) } } }
  r.report("set include") { TIMES.times { access.each {|v| set_index.include?(v) } } }
end

The output (ruby 1.9.1p378 (2010-01-10 revision 26273) [x86_64-linux]):

                   user     system      total        real
array index    0.730000   0.000000   0.730000 (  0.745341)
hash index     0.990000   0.000000   0.990000 (  1.003517)
hash include   1.310000   0.000000   1.310000 (  1.332846)
set include    1.970000   0.000000   1.970000 (  2.024808)

It's faster to numerically round than to sprintf in ruby

require 'benchmark'

class Float
  def round_to(x)
    (self * 10**x).round.to_f / 10**x
  end

  def ceil_to(x)
    (self * 10**x).ceil.to_f / 10**x
  end

  def floor_to(x)
    (self * 10**x).floor.to_f / 10**x
  end

  def round_to_sprintf_f(x)
    ("%.#{x}f" % x).to_f
  end

  def round_to_sprintf_sprintf(x)
    sprintf("%.#{x}f", x)
  end

  def round_to_sprintf_modulus(x)
    "%.#{x}f" % x
  end

end

num = 34.2342134
place = 3

TIMES = 1000000
Benchmark.bmbm do |r|
  r.report("numeric round_to (float)") { TIMES.times { num.round_to(place) } }
  r.report("numeric round_to (to string)") { TIMES.times { num.round_to(place).to_s } }
  r.report("sprintf (float)") { TIMES.times { num.round_to_sprintf_f(place) } }
  r.report("sprintf (string)") { TIMES.times { num.round_to_sprintf_sprintf(place) } }
  r.report("sprintf modulus (string)") { TIMES.times { num.round_to_sprintf_modulus(place) } }
end

The output:

% ruby -v
ruby 1.9.1p378 (2010-01-10 revision 26273) [x86_64-linux]
...
                                   user     system      total        real
numeric round_to (float)       0.820000   0.000000   0.820000 (  0.819920)
numeric round_to (to string)   1.990000   0.000000   1.990000 (  1.989912)
sprintf (float)                3.620000   0.000000   3.620000 (  3.614758)
sprintf (string)               2.650000   0.000000   2.650000 (  2.654812)
sprintf modulus (string)       3.190000   0.000000   3.190000 (  3.191807)

The conclusion: Regardless of whether you are going to a string or numeric after rounding, it is faster to round numerically than with sprintf. Also, using the sprintf function is faster than '%'. Usually speed doesn't matter, but when it does...

apply & sweep : Column-wise & Row-wise operations in R

How do you do operations on rows and columns in R? apply and sweep are the main tools. Here is an example of "autoscaling" (mean center and divide by standard deviation) on each column:

[I verified that this is correct on oocalc]

# 1 -> across the rows
# 2 -> across the columns 
m <- matrix(data=c(1,2,3,4,4.5,6,7,8,9,10,11,13), nrow=4, ncol=3)
m
#      [,1] [,2] [,3]
# [1,]    1  4.5    9
# [2,]    2  6.0   10
# [3,]    3  7.0   11
# [4,]    4  8.0   13

colmeans <- apply(mymatrix, 2, mean)  # the column-wise means
# mc = the column-wise mean centered data
mc <- sweep(m, 2, colmeans, "-")   # subtract is the default

col_stdev <- apply(m, 2, sd)  # column-wise standard deviations

mcstd <- sweep(mc, 2, col_stdev, "/")  # divide by standard deviations

mcstd
#            [,1]       [,2]      [,3]
# [1,] -1.1618950 -1.2558275 -1.024695
# [2,] -0.3872983 -0.2511655 -0.439155
# [3,]  0.3872983  0.4186092  0.146385
# [4,]  1.1618950  1.0883839  1.317465

Regression line, R^2 (Pearson's correlation coefficient), slope, and y intercept in R

A basic feature in Excel and OpenOffice Calc is to find and plot the regression line. Here is an example of doing this in R. This is more work in R, but you are sacrificing for more power.

Here is how we get the data out:

x = c(1,2,3,4,5)
y = c(3,4,5,5.1,6.2)
pearsons_r = cor(x,y)
r_squared = pearsons_r^2
fit = lm(y~x) # notice the order of variables
y_intercept = as.numeric(fit$coeff[1])
slope = as.numeric(fit$coeff[2])

Here is how we plot it:

plot(x,y)
abline(lm(y~x)) # again, notice the order
function_string = paste("f(x) = ", slope, "x + ", y_intercept,  sep="")
r_sq_string = paste("R^2 =", r_squared)
display_string = paste(function_string, r_sq_string, sep="\n")
mtext(display_string, side=3, adj=1)  # top right outside of the margin