JP van Oosten

Custom Jinja2 filters when using bottle

2013-03-24T00:00:00+01:00

I have been experimenting with Bottle lately. I have done a bit of Django development and I’ve always liked its template engine, especially the idea of template inheritance and the fact that it’s not a layer on top of xml. So, when starting to experiment with Bottle, I wanted to use Jinja2 as that was inspired by the Django template engine.

One of the nice features of Jinja2 (and its inspiration) are filters. For example, in a template, one can use {{ name|capitalized }} which will output the value of name, but then capitalized. As this is merely for presentation, I think this actually belongs in the template.

You can even write your own filters, which are basically just python functions that are being called by the parser. However, I could not really find a clear explanation of how to automatically use my custom filters in the @view(templatename) decorator offered by bottle. Here is the solution I came up with:

from bottle import jinja2_view
view = functools.partial(jinja2_view,
	template_settings={'filters': filter_dict})

where filter_dict is a dictionary with a name to function mapping. functools.partial is a nice function (that is actually used in the bottle definition of jinja2_view as well), that allows you to wrap a function such that certain (keyword) arguments already filled in. In other words: the above code creates a new function that calls jinja2_view with the template_settings argument already filled in. Now, you don’t have to pass template_settings each time you call view.

To automate adding filters to the filter_dict as much as possible, I wrote a simple @filter decorator:

filter_dict = {}
def filter(func):
	"""Decorator to add the function to filter_dict"""
	filter_dict[func.__name__] = func
	return func

# Usage:
@filter
def quote(s):
	return urllib.quote(s)

Notes from the Debian packaging symposium

2012-12-22T00:00:00+01:00

A few weeks ago, Frank Brokken and Jurjen Bokma organised a Debian / Ubuntu packaging symposium, with special guest Tony Mancill. We learned a lot about creating Debian packages and creating repositories. I wrote this blog-post to make sure I don’t forget the most important steps and maybe give others a chance to learn how to create packages as well.

These notes are a combination of my own experiences and notes that can be found on the symposiums website. Please don’t hesitate to contact me if you have any questions or remarks (jp at this domain).

Quick overview (or: TL;DR)

Create COW chroot environment
Configure pbuilder and git-buildpackage
Run dh_make to create the required debian/ directory
Fill in the required fields in debian/control
Run either pdebuild or git-buildpackage

Required packages

Install the following packages:

build-essential
dh-make
devscripts
cowbuilder
pbuilder
git-buildpackage
debhelper

Chroots and cowbuilder

It is a good idea to build your packages in a chroot. Besides giving you a clean and homogeneous environment, it is a convenient way to test whether your package builds and installs in a clean environment (i.e., all your dependencies are correctly specified).

Cowbuilder is a tool to create “copy-on-write” chroot environments. This means that when you work in the chroot environment (building or testing your package), files that are changed are cleaned-up after you’re finished. This means that you can set-up your clean environment once, and be confident it stays clean. The advantage over copying the entire environment everytime is that files are only copied when they are being written to.

In order to create your cowbuilder environment, you can either use the cowbuilder-create-base script (mirror) Jurjen created, or use the following:

sudo cowbuilder --create --distribution=sid \
	--basepath=/var/cache/pbuilder/base-sid.cow

Make sure that, if you use Jurjens script, you create a symlink from /var/cache/pbuilder/$DIST-base.cow to /var/cache/pbuilder/base-$DIST.cow because some build-tools expect the COWs to be at the latter location.

Tony suggested a couple of shell-functions for easy entering and saving changes in chroots:

cb-shell () {
	chr=$1 ; shift
	sudo cowbuilder \
		--bindmount $HOME \
		--login \
		--basepath=/var/cache/pbuilder/base-${chr}.cow $@
}

cb-shell-save () {
	cb-shell $@ --save-after-login
}

The cb-shell-save function can be used to adjust your chroot a bit. Good practice is to put the name of your chroot in /etc/debian_chroot, so that it will appear in your prompt (which is convenient to quickly see whether you are working in your chroot or not). You can also create a local user, which persists if you use cb-shell-save.

Configuring your environment

In this step, we will create the configuration files and environment variables necessary for the debhelper-scripts, pbuilder and git-buildpackage.

First, make sure you have a valid gpg key. If you don’t have one, create one with gpg --gen-key. It’s good practice to actually use a (long) phrase as your passphrase. Since you must type it a couple of times, you can use an agent for convenience. Install gnupg-agent and make sure use-agent is specified in ~/.gnupg/gpg.conf if you want to use the agent. Make sure you also put the following in your shell start up scripts (.bashrc or so):

if test -f $HOME/.gpg-agent-info && kill -0 $(cut -d: -f 2 $HOME/.gpg-agent-info) 2>/dev/null; then
	eval $(cat $HOME/.gpg-agent-info)
else
	eval $(gpg-agent --daemon --write-env-file $HOME/.gpg-agent-info)
fi
GPG_TTY=$(tty)
# GPG_AGENT_INFO is set from within $HOME/.gpg-agent-info, but still needs to be exported
export GPG_TTY GPG_AGENT_INFO

For dh_make and the build-scripts, you need to specify the following environment variables:

export DEBEMAIL=john@sudoe.com
export DEBFULLNAME="John Sudoe"
export DEBSIGN_KEYID=D1D9258A

DEBSIGN_KEYID is the id of the gpg key with which you want to sign your packages. You can find it using gpg --list-secret-keys.

Note that if you use git, it is a good idea to make sure you use the same name and email address to commit your debian-changes. Put the following in your ~/.gitconfig or in <project>/.git/config:

[user]
		name = John Sudoe
		email = john@sudoe.com

Next, we will configure pbuilder. Its configuration file, ~/.pbuilderrc, is a shell-script. Jurjen suggests the following (I changed ${DIST}-base.cow to base-${DIST}.cow to conform to pbuilder defaults):

PDEBUILD_PBUILDER=cowbuilder

if [ -n "${DIST}" ]; then
	export DISTRIBUTION=$DIST

	AUTO_DEBSIGN=yes

	BASETGZ="$(dirname $BASETGZ)/$DIST-base.tgz"

	export BASEPATH=/var/cache/pbuilder/base-${DIST}.cow
	export BUILDPLACE=/var/cache/pbuilder/${DIST}.cow

	install -d ../lastbuild/${DIST}
	export BUILDRESULT=../lastbuild/${DIST}

	if echo $@|grep -qe --create ; then
		BUILDPLACE=/var/cache/pbuilder/base-${DIST}.cow
		install -d ${BUILDPLACE}
	fi

	APTCACHE="/var/cache/pbuilder/$DIST/aptcache/"

	case ${DIST} in
		'lenny'|'squeeze'|'wheezy'|'sid' )
			MIRRORSITE=http://ftp.nl.debian.org/debian
			;;
		'lucid'|'maverick'|'natty'|'oneiric'|'precise' )
			MIRRORSITE=http://nl.archive.ubuntu.com/ubuntu
			;;
	esac

fi

And finally, we configure git-buildpackage by putting the following in ~/.gbp.conf:

[DEFAULT]
builder = /usr/bin/git-pbuilder
cleaner = fakeroot debian/rules clean
postbuild = lintian $GBP_CHANGES_FILE
pristine-tar = True
upstream-branch = upstream

[git-buildpackage]
export-dir = ../build-area/
tarball-dir = ../tarballs/

[git-import-orig]
dch = false

Building a package

There are a number of ways to build a package. If you only have a source tar-ball, you can use dh_make and pdebuild. The first tool creates the debian/ directory, with debian/rules – specifying how to build the package – and debian/control – specifying the name, license, section, dependencies and description, among others. pdebuild is a wrapper around a couple of other wrappers: pdebuild uses debuild, which in turn uses dpkg-buildpackage, which use the dh_* scripts to run ./configure, make and make install. pdebuild also can use cowbuilder to run everything in the copy-on-write chroot to keep your environment clean.

Example

The following example is from the exercises at the symposium.

wget http://gnu.xl-mirror.nl/hello/hello-2.7.tar.gz
tar zxf hello-2.7.tar.gz
cd hello-2.7/
dh_make -f ../hello-2.7.tar.gz -s -c gpl3 && rm debian/{*.ex,*.EX,README.*}
DIST=sid pdebuild

The dh_make line uses -f ../hello-2.7.tar.gz to specify the location of the original source archive, -s to indicate we build a single binary, and -c gpl3 to specify the gpl3 license. After that, we remove the example and readme files that we don’t need. Finally, we build the package for the sid distribution. This is done inside a clean copy of the base-sid.cow chroot.

Building packages from git repositories

When creating debian packages from upstream sources, you can use git to keep track of debian specific changes. A typical layout of your repository would be to have an upstream branch, containing the actual upstream sources; a pristine-tar branch, used to (re)create tar balls exactly (see below) and a master branch, combining upstream with your debian/ directory. With this layout, building the package can be done quickly with DIST=sid git-buildpackage. If you have configured git-buildpackage as above, then your package will be located in ../build-area.

git-buildpackage expects to have <package>_<version>.orig.tar.gz in ../tarballs/. A good way to keep the tarballs in your repository is using pristine-tar, which commits a tarball and subsequent delta’s so that you don’t need to store complete tarballs for each release. Putting a tarball in version control can be done with pristine-tar commit <package>_<version>.orig.tar.gz. Creating it for use with, for instance, git-buildpackage, can be done with pristine-tar checkout ../tarballs/<package>_<version>.orig.tar.gz. The advantage is that you can keep copies of the tarballs, but don’t have to waste space for files that don’t change between versions.

Final remarks

These are the basic steps to take when creating debian packages. The COWbuilder set-up is really convenient, especially if you create and test packages in a number of different settings (for example, you can have a default chroot for sid and wheezy and any Ubuntu release). Using git and git-buildpackage, you can keep track of your changes to the debian-specific files and automatically have it generate changelogs for you (using git-dch).

If you want these packages to appear in the debian repositories, you will probably have to contact someone willing to sponsor your upload. To get a sponsor, you usually have to make a request on the debian-mentors mailinglist (see the debian mentors FAQ for more information).

If you would rather host your own repository, we briefly touched on how to manage repositories with reprepro, but that is a topic for another post.

If you read this far, and didn’t have your question answered, or have remarks about the post, you are welcome to e-mail me at jp at this domain, or contact me through twitter @jpvoosten.

FBB software (bobcat & bisonc++) on the mac

2011-07-23T00:00:00+02:00

Installing software by Frank Brokken (FBB) on the mac used to be a bit of a hassle. The build software is somewhat tailored to Linux (specifically Debian). However, with homebrew installation on the mac is now almost as convenient as on Debian. I have prepared some formulae (that’s what homebrew calls packages) for icmake, yodl, bobcat and bisonc++.

There are two ways to install these packages: either by providing the URL of each formula on the command line to brew install (in the correct order), or by downloading the formula in /usr/local/Library/Formula (or whatever brew --prefix tells you instead of /usr/local) and then issuing brew install bisonc++.

Either way, you will first have to install a recent g++. You can find a formula for gcc4.6 on github (mirror), which you can install with:

brew install --use-gcc --enable-cxx https://raw.github.com/Sharpie/homebrew-science/ceb4de17c5b44fc4570cd86b4cf069d3eaa89698/duplicates/gcc.rb

The --use-gcc option is required because by default homebrew will try to compile using the llvm-gcc compiler. The --enable-cxx makes sure g++-4.6 will also be installed. Compiling gcc will take some time though. On my machine (a MacBook, 2.2Ghz Core 2 Duo) it took about 50 minutes.

Then you can proceed to install Frank’s packages:

brew install http://jpvanoosten.nl/homebrew/formula/icmake.rb
brew install http://jpvanoosten.nl/homebrew/formula/yodl.rb
brew install http://jpvanoosten.nl/homebrew/formula/bobcat.rb
brew install http://jpvanoosten.nl/homebrew/formula/bisonc++.rb

cd `brew --prefix`/Library/Formula
wget http://jpvanoosten.nl/homebrew/formula/{icmake,yodl,bobcat,bisonc++}.rb
brew install bisonc++

Either way, you can find my formula to Frank’s packages here:

After installation, every package is installed into it’s own ‘keg’ in /usr/local/Cellar/ and symlinked into /usr/local/bin. With homebrew you can easily remove the packages if needed.

Hope this is useful for somebody. If you have any comments or questions, you can reach me by the address found below, or just poke me on twitter.

(Thanks to Jelmer, Thieme and Jakob for reporting some issues with the Formulae. They should be fixed now.)

Can Markov properties be learned by hidden Markov modelling algorithms?

2010-09-21T00:00:00+02:00

Recently I received my MSc degree in Artificial Intelligence. The topic of my final project and thesis was hidden Markov models (HMMs). The main questions were whether the properties of Markov processes could be reliably learned by the current hidden Markov modelling algorithms and whether the Markov property is important in language and handwriting recognition.

Part of the thesis is an in-depth overview of the theory of hidden Markov models and the Baum-Welch algorithm, so if you are interested in HMMs in general, or in the conclusions of the research, please feel free to read my thesis: Can Markov properties be learned by hidden Markov modelling algorithms?

The abstract:

Hidden Markov models (HMMs) are a common classification technique for time series and sequences in areas such as speech recognition, bio-informatics and handwriting recognition. HMMs are used to model processes which behave according to the Markov property: The next state is only influenced by the current state, not by the past. Although HMMs are popular in handwriting recognition, there are some doubts about their usage in this field.

A number of experiments have been performed with both artificial and natural data. The artificial data was specifically generated for this study, either by transforming flat-text dictionaries or by selecting observations probabilistically under predefined modelling conditions. The natural data is part of the collection from the Queen’s Office (Kabinet der Koningin), and was used in studies on handwriting recognition. The experiments try to establish whether the properties of Markov processes can be successfully learned by hidden Markov modelling, as well as the importance of the Markov property in language in general and handwriting in particular.

One finding of this project is that not all Markov processes can be successfully modelled by state of the art HMM algorithms, which is strongly supported by a series of experiments with artificial data. Other experiments, with both artificial and natural data show that removing the temporal aspects of a particular hidden Markov model can still lead to correct classification. These and other critical remarks will be explicated in this thesis.

Improve your writing with shell scripts

2010-07-26T00:00:00+02:00

Currently, I’m working on my Master’s thesis on Hidden Markov Models. Matt Might wrote an article on three shell scripts to improve your writing, which I found interesting. The scripts help to detect the use of passive voice, weasel words (such as “surprisingly low”) and duplicate words (which are difficult to detect when a line break separates them).

One of the remarks I repeatedly received was that my paragraphs were much too short. A couple of paragraphs were just one or two sentences, which I could usually just throw together.

In the light of the Matt’s scripts, I wrote my own version, in which I detect paragraphs with only two or three sentences and spanning only a few lines. It isn’t perfect and not all small paragraphs need to be long, but it might warrant a closer inspection of your text.

#!/usr/bin/env python
import sys
import re

SINGLE_COMMAND_RE = re.compile(r'^\\\w+\{[^}]+\}$')

def process(file):
	"""Ignores lines containing only a single command at the beginning of a
	paragraph (piece of text surrounded by blank lines)."""
	paragraph = [0, 0, 0] # [start, sentence count, linecount]
	prev_line = None
	for linenum, line in enumerate(file):
		line = line.strip()
		if SINGLE_COMMAND_RE.match(line) and not prev_line:
			continue
		if not line and paragraph[1]:
			report_short_paragraph(filename, paragraph)
			paragraph = [0, 0, 0]
		else:
			if paragraph[0] == 0:
				paragraph = [linenum, 0, 0]
			paragraph[1] += line.count('.')
			paragraph[1] -= 2 * line.count('...')
			paragraph[2] = linenum - paragraph[0] + 1
		prev_line = line

def report_short_paragraph(filename, paragraph):
	if paragraph[1] <= 2 and paragraph[2] < 4:
		print '%s:%d paragraph of %d sentence(s) / %d lines' % (filename,
				paragraph[0] + 1, paragraph[1], paragraph[2])

if __name__ == "__main__":
	for filename in sys.argv[1:]:
		file = open(filename)
		process(file)

Creating MoinMoin pages programmatically

2010-05-13T00:00:00+02:00

Recently, I installed a MoinMoin wiki and a mailing list using mlmmj for a group of people, and I wanted to create a restricted page on the wiki with the subscribers of the list.

Since this mailing list is not owned by the user the wiki runs as, the script had to be executed as root. As soon as the subscribers were collected, the effective user id was changed with a simple call to os.seteuid. This also meant I could not create a macro and have the page update from within the wiki itself.

The solution I found was to create a user which was privileged to edit a page with ACLs. The MoinMoin wiki has a page which shows you how to edit pages programmatically, but for some reason I also had to set the valid property of the user to 1 (MoinMoin seems to use valid = 0 by default, not False).

It felt a bit like a hack, but it worked. As a nice bonus, the PageEditor.saveText method throws a PageEditor.Unchanged exception if the page is not changed, so only a new revision is filed when the page contents actually do change.

LaTeX font installation messes up Mac OS X font cache

2010-02-25T00:00:00+01:00

A few days ago, I tried to install a new font for LaTeX from an OTF-file. I used the script otfinst.py, by John Owens. However, when I decided not to use the font after all, none of the regular LaTeX fonts (Computer Modern, by Knuth for example) seemed to work any more, everything was replaced by an ugly sans serif font.

Thinking I must have messed up LaTeX, I set out to reinstall and undo all the changes I made. This was actually not the case; after searching the web a bit I found that sometimes the font cache would get corrupted. To interact with the font cache in Leopard (Mac OS X 10.5), one uses the atsutil command:

atsutil databases -removeUser # clears the cache
atsutil server -shutdown      # restart the ATS server
atsutil server -ping          # see when the server is back up again

You might want to issue the first command as root to clear the cache of all users.

Be sure to restart any running programs such as Safari and Preview, or you will still see weird fonts.

See this article on Macworld for more information.

Gitosis: arguments to command look dangerous

2009-11-12T00:00:00+01:00

If you get the following error when trying to execute git pull: “Arguments to command look dangerous”, then you might want to check whether the remote repository ends with a slash (/). This tripped me and a friend over the other day. Since gitosis is not very verbose with its error-messages, I had to dig deep to find this out.

In general, the path to the git repository needs to start with an alphanumeric character followed by zero or more alphanumeric characters, @, _ or -. Subdirectories have the same restrictions. So, paths cannot end in a slash.

Just edit the corresponding remote section (usually this is origin and the section looks like [remote "origin"]) in the file .git/config and correct the path after the colon (:), for example by removing the slash at the end.

authn_dbd error "User not found"

2009-11-05T00:00:00+01:00

If you use apache and the DBD module for authentication, you need to be very careful with using quotes in your password query.

I had the following configuration:

AuthBasicProvider dbd
AuthDBDUserPWQuery "SELECT password FROM users WHERE name='%s'"

but this yielded the very verbose message: “user user not found”. After a while a friend commented he used the same query, minus the quotes around %s.

So, apparently, the authn_dbd module does the correct quoting for you. Use the following query if you encounter this behaviour:

AuthBasicProvider dbd
AuthDBDUserPWQuery "SELECT password FROM users WHERE name=%s"

Python distutils woes

2009-10-21T00:00:00+02:00

Today I was working on a setup script for submin, but I got some unexpected results while testing. Apparently, I had messed up a path by using string concatenation (pro-tip: always use os.path.join).

Every time I tried something else, the same faulty paths were copied to the install directory. How did I fix it? By removing the build/ directory.

So, if you have weird problems with distutils, try removing the build/ directory. This directory contains the files which are copied to the final installation directory, and is not cleaned up properly after finishing the install command.

Stripping file extensions in C

2009-09-14T00:00:00+02:00

A very simple C-function to strip all extensions from a filename. The function removes everything from the first dot.

void strip_extension(char *name)
{
	while (*++name != '.' && *name != 0)
		;
	*name = 0;
}

C++ expected initializer

2009-04-13T00:00:00+02:00

When I was developing a new feature for Flexc++, I ran into a compiler error stating “expected initializer before ‘const’”.

I was building a function which returned an object of an inner class. However, the outer class was a class template, and the template parameter was used in the inner class:

template <typename StreamInfo>
class ScannerTemplate
{
	protected:
	struct BufferData
	{
		StreamInfo d_streamInfo;
	};
	public:
	BufferData const switchBuffer(StreamInfo const &streamInfo);
};

The implementation of switchBuffer looked like this:

template <typename StreamInfo>
inline ScannerTemplate<StreamInfo>::BufferData const       
	ScannerTemplate<StreamInfo>::switchBuffer(StreamInfo const &streamInfo)
{
	// ...
}

Why did I get the error that an initializer was expected?

After some digging, I found that this is a (deprecated) implicit typename: At compile time, the compiler does not know what the instantiation of ScannerTemplate looks like.

To turn this into an explicit typename, just prefix typename to the return type:

template <typename StreamInfo>
inline typename ScannerTemplate<StreamInfo>::BufferData const       
	ScannerTemplate<StreamInfo>::switchBuffer(StreamInfo const &streamInfo)
{
	// ...
}

For more information, look in the C++ Annotations as usual :)

istringstream::str not resetting eof

2008-10-27T00:00:00+01:00

Today I was implementing a c++-function which read single words from a file. Because the first few characters of each line needed to be stripped, I could not use simple string extraction from the filestream. This lead me to use the istringstream.

Unfortunately, the istringstream gave me a minor problem: it only seemed to extract words for the first line in each file!

Because I wanted to reuse a single stream, I used the istringstream:: str(string const &text) method to fill the stream with the current line’s content. From there I used the string extraction.

After a bit, I remembered I had seen this problem before, and remembered I needed to reset something. After a bit of digging, I found out that when the extraction reached the end of the string, the eof-bit would be set. Filling the stream with new data using the str method, did not reset these bits, unfortunately.

The fix was then easy: istringstream::clear() does the job.

FOAF

2008-09-04T00:00:00+02:00

For a course on Semantic Web Technology, we needed to create a FOAF-file.

You can use the link below, or you can use the link in the head-portion of my website.

This is my entry, you can add me if you like: http://jpvanoosten.nl/s/foaf.rdf

tuple vs list

2008-08-04T00:00:00+02:00

Everyone with a bit of python knowledge can intuitively feel that tuples are faster than lists. Of course: tuples are unmutable, and therefore should be faster. I was wondering today, how much faster tuples were.

Again, we use the timeit module to measure runtime of a command.

$ python -m timeit '[True]'
1000000 loops, best of 3: 0.206 usec per loop

$ python -m timeit '(True,)'
10000000 loops, best of 3: 0.118 usec per loop

So, using a tuple in such a simple case is almost 1.8 times faster!

(I like the fact that I can test my intuition with timeit)

SumQuerySet

2008-07-18T00:00:00+02:00

When implementing the model for a timetracking application, I was trying to sum the amount of hours for a certain queryset. Not succeeding by digging through the internals of django, I tried this – very simple – approach. It uses a custom Manager, which in turn uses a custom QuerySet.

At first, I tried to use the SQL-function SUM, but it was hard to tell the Django-ORM to use this standard SQL-function. I then decided to let python do it’s job instead.

The model has a field called hours, in my case a CharField (because I wanted to save time information in a “2:45” kind of format).

A custom Manager is used to return a queryset:

class SumManager(models.Manager):
    def get_query_set(self):
        return SumQuerySet(self.model)

Nothing fancy. get_query_set just returns a custom SumQuerySet:

class SumQuerySet(QuerySet):
    def sum_hours(self):
        return sum(float(entry.hours) for entry in self)

As you can see, this queryset has an additional method sum_hours, which is very simple. It just returns the sum of all entry.hours in the current queryset. This enables one to use a filter and then ask the queryset to return the sum of all the hours.

Then add the line objects = SumManager() to the model and all’s set!

Bonus:
You can also change the pagination-template to show the amount of hours in the current view in the admin:

{% load admin_list %}
{% load i18n %}
<p class="paginator">
{% if pagination_required %}
{% for i in page_range %}
	{% paginator_number cl i %}
{% endfor %}
{% endif %}
{{ cl.result_count }} {% ifequal cl.result_count 1 %}{{ cl.opts.verbose_name|escape }}{% else %}{{ cl.opts.verbose_name_plural|escape }}{% endifequal %}
{% if cl.query_set.sum_hours %}
&mdash; {{ cl.query_set.sum_hours }} hours total
{% endif %}
{% if show_all_url %}&nbsp;&nbsp;<a href="{{ show_all_url }}" class="showall">{% trans 'Show all' %}</a>{% endif %}
</p>

This shows, if the sum_hours is available in the current queryset, the number of hours of the records shown. (For this to work, you need to tell the admin you want to use your own SumManager as the default manager. Add the line manager = SumManager() to your Admin-inner class.)

Flexc++ textmate language

2008-06-18T00:00:00+02:00

While working on our replacement for flex / flex++, I was getting lost in all the comments and regular expressions in our lexer-file. I needed syntax highlighting badly.

So, I opened up the bundle editor and wrote a language-file for flexc++. Because it’s basically the same syntax, it also works on lexers for flex.

If someone’s interested, drop me a line at jp@thisdomain, and I’ll send the bundle to you (to save you some cut’n’paste work :)

{	scopeName = 'source.flexc++';
	fileTypes = ( );
	foldingStartMarker = '/\*\*|[^%]\{\s*$';
	foldingStopMarker = '\*\*/|^\s*\}';
	patterns = (
		{	name = 'comment.block.flexc++';
			begin = '\s*/\*';
			end = '\*/\s*';
		},
		{	name = 'meta.codeblock.flexc++';
			begin = '%\{';
			end = '%\}';
			patterns = ( { include = 'source.c++'; } );
		},
		{	name = 'constant.other.namedefinition.flexc++';
			match = '^\s*[A-Z_0-9]+';
		},
		{	name = 'string.regexp.namedefinition.flexc++';
			match = '(?<=[A-Z_0-9])\s+.+$';
		},
		{	name = 'meta.rulessection.flexc++';
			begin = '^%%$';
			end = '%%';
			patterns = (
				{	name = 'meta.blankline.flexc++';
					match = '^$';
				},
				{	name = 'comment.block.rulessection.flexc++';
					begin = '\s*/\*';
					end = '\*/\s*';
				},
				{	name = 'entity.name.other.startconditionlist.flexc++';
					match = '^\s*<[a-zA-Z_,]+>';
				},
				{	name = 'string.regexp.flexc++';
					begin = '^\s*(?=[^\s\n])|(?<=>)(?=[^\s])';
					end = '(?=\s|\n)';
					patterns = (
						{	name = 'string.regexp.characterclass.flexc++';
							match = '\[.+\]';
						},
						{	name = 'string.regexp.escape.flexc++';
							match = '\\.';
						},
						{	name = 'string.regexp.string.flexc++';
							begin = '"';
							end = '"';
							patterns = (
								{	name = 'string.regexp.string.escapechar.flexc++';
									match = '\\.';
								},
							);
						},
						{	name = 'constant.other.name.flexc++';
							match = '\{[A-Z_0-9]*[A-Z_][A-Z_0-9]+\}';
						},
					);
				},
				{	name = 'meta.actionblock.flexc++';
					begin = '\{';
					end = '\}';
					patterns = (
						{	name = 'entity.name.function.builtin.flexc++';
							match = '\b(BEGIN)\b';
						},
						{	name = 'meta.actionblock.source.flexc++';
							include = 'source.c++';
						},
					);
				},
				{	name = 'entity.function.builtin.flexc++';
					match = '\b(BEGIN)\b';
				},
				{	name = 'meta.action.flexc++';
					contentName = 'source.c++';
					match = '(?<=[^\s\n])\s+.+$';
					include = 'source.c++';
				},
			);
		},
		{	name = 'entity.other.attribute-name.flexc++';
			match = '^%[^ ]+';
		},
	);
}

More on list.sort

2008-06-16T00:00:00+02:00

As mentioned in the previous post, I found a new way to call list.sort in python. I did some research with the timeit-module and these were my findings.

Using the cmp-method, the timeit module returned 0.008760 sec/pass, while using the key-method gave me 0.001954 sec/pass. Wow, what a difference!

Here is the code I used:

import timeit
import random

class A(object):
	def __init__(self, data):
		self.data = data
	def __repr__(self):
		return "A(%d)" % self.data

l = [A(x) for x in xrange(1000)]
def get_random_list():
	global l
	random.shuffle(l)
	return l

s = """import operator; get_random_list().sort(lambda a,b: cmp(a.data, b.data))"""
t = timeit.Timer(stmt=s, setup="from __main__ import get_random_list")
s2 = """import operator; get_random_list().sort(key=operator.attrgetter('data'))"""
t2 = timeit.Timer(stmt=s2, setup="from __main__ import get_random_list")

ntries = 10000
print "cmp-method: %f sec/pass" % (t.timeit(number=ntries)/ntries)
print "key-method: %f sec/pass" % (t2.timeit(number=ntries)/ntries)

Combining queries in Django

2008-06-16T00:00:00+02:00

While implementing the code for this blog, I was wondering wether I should implement a “Links” application, where I would be able to add links I thought interesting. While investigating a method to combine the results for both this and the Post models, I found an interesting little feature in Python: operator.attrgetter.

So, if I have a list containing results from the two models (obtained through, for example, itertools.chain) I can sort them like this:

a.sort(key=operator.attrgetter('date'))

It seems that this is more efficient on large datasets, than using

a.sort(lambda a,b: cmp(a.date, b.date))

but more interestingly, I think this is more in line with the way the STL works in C++.

import operator my new friend

(Oh, if you have two querysets for the same model, it is obviously more efficient to use q1 | q2.)

Perl oneliner to sum all lines

2008-06-15T00:00:00+02:00

As I’m not such a big Perl-star, here is my first oneliner! This one sums all integers, presented at separate lines.

perl -e 'for (<STDIN>) { $sum += $_; } print $sum . "\n";'

(I didn’t know I could leave the $sum = 0; out, but someone with more perl knowledge pointed this out to me)

Very handy!