Pattern Recognition: December, 2007

<< First < Previous Next > Last >>

December 09

My first conference paper

I was excited to hear that the paper "Finding parallel texts on the web using cross-language information retrieval" authored by myself and Fei Xia was accepted for the cross-language information access workshop at the IJCNLP 2008 conference! I won't go to Hyderabad myself, but hope our results will help other researchers build parallel corpora. And if you are not a language researcher: "parallel corpora" is just a fancy term for translated texts in two languages (preferably lots of them).

8:35 PM | Add a comment | Permalink | Blog it | Computational Linguistics

Perl script to detect encoding of one or multiple text files

Recently I had the need to do a very common task again: identify the character encoding of a bunch of text files in a directory. The tedious way to do this is to open each file in an editor that detects the encoding and then see what encoding was identified (e.g. opening it in Notepad and choosing File/Save As ...). Of course one would think that there are command line tools available online that achieve just that on a number of files without opening them manually. My searches always turn up descriptions how character encoding detection is done in the browsers or library functions for the purpose. But no command line tools!? Finally I got fed up and took some time to put together a short Perl script:

#!/usr/bin/perl -w

use strict;
use Encode::Guess;

if(@ARGV != 1)
{
    print "Usage: perl $0 <file to analyze>\n";
    exit;
}

my @files = glob($ARGV[0]);

foreach my $file (@files)
{
    open(FILE,$file);
    binmode(FILE);
    if(read(FILE,my $filestart, 500)) {
	my $enc = guess_encoding($filestart);
	if(ref($enc)) {
	    print "$file:\t",$enc->name,"\n";
	}
	else {
	    print "Encoding of file $file can't be guessed\n";
	}
    }
    else {
	print "Cannot read from file $file\n";
    }
}

close(FILE);

This assumes you have Perl installed (tested on v5.8.8) including the Encode::Guess module. This is not very sophisticated (see the documentation for the module), but is sufficient for most jobs, e.g. quickly finding out in what encoding somebody has saved some files or if the files are all in a consistent encoding.