Flameeyes's Weblog

Blast from the recent past; two years ago

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Wed, 02 Jul 2008 23:36:00 +0000

Today I was feeling somewhat blue, mostly because I’m demotivated to do most stuff, and I wanted to see what it was like to work in Gentoo two years ago.

One thing I read is that a little shy of exactly two years ago, ICQ broke Kopete, just like they did yesterday. Interestingly enough, even though a workaround has been found for Kopete 0.12 (the one shipped with KDE 3.5), there is no bump I see in the tree this time. Sign that the KDE support in Gentoo has changed, most likely.

There is also the whole thing with ALSA problems, which span so many posts that it’s not worth listing all them. The current ALSA maintainer simply gave up on providing something that, at least for some users, ended up being quite important.

And all the work on Gentoo/FreeBSD! Although Javier is doing a huge work now to support FreeBSD 7.0, he’s not prone to blog about it, and you can see that Gentoo/FreeBSD is easily ending up in the “historical” memory, rather than being discussed and tried out by users daily.

What didn’t change at all is my insomnia, it’s almost 2AM and I’m still up. And this time I don’t even have Antiques Roadshow to watch. I’m currently working on xine, just like two years ago.

In general, I think a lot of areas in Gentoo did go downhill from two years ago, rather than improving. While Portage is certainly improved, thanks to Zac, Genone and the rest of the team, and we can see that in the new extended repoman checks, that also helps QA. But the general user support seems, to me, lacking.

This is a direct consequence, in my opinion, of leaving open doors for people who are just driving Gentoo’s energy away, by taking over projects to make them stall, by discussing details over and over and over, by repeating the same request even when people reject it as it stands, and so on.

I hope things will improve in the next months, thanks to a new council that can finally grow some balls, straightening up the situation, but if this does not happen, I’m already preparing for my plan B…

Again about .la files (or why should they be killed off sooner rather than later)

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Wed, 02 Jul 2008 13:43:00 +0000

I’ve been working as an experiment on rewriting xclip to use XCB rather than Xlib. This is mostly because I always have been interested in XCB but I never had time to learn the internals too much.

To make my task easier I ended up using some funcitons that are not available in the currently-released version of xcb-util, the side-package of XCB that contains some higher-level functions that make it easier to replace Xlib.

Beside the fact that xcb-util still haven’t bumped its version, which makes it impossible to check for the right version with pkg-config, there is one interesting point in using the latest available version through the x11 overlay.

Letting alone some problems with being able to actually fetch and install the packages I need (Donnie, I’ll send you the patches later if I can polish them a bit), over the actual GIT tree there are a few patches applied, coming from Jamey Sharp (an XCB developer) from March 2008. These remove one library (libxcb-xlib) and change the locking method used to make Xlib use the same socket as XCB. These changes not only break ABI (without changing the soname, alas!) but also make it impossible to build the old libX11 against the new libxcb. Using the live version of libX11 (that is also patched to use the new hand-out mechanism) fixes this problem, but the result is a way bigger trouble.

First of all, this is a perfectly good example of what I said about preserve-libs. If you are not using --as-needed, and you had libX11 built with xcb USE flag enabled, you’ll have libxcb.so.1 links on almost all X-using binaries in your system; after rebuilding the new libxcb and libX11 (which respectively would install libxcb.so.2, in theory, and let libX11 link to that), all the binaries will have in their process space both the old and the new libxcb. With different ABIs. And that’s a huge problem on itself.

Then there is the other problem, that is related to the .la files I discussed a few months back. As a huge amount of KDE modules (and not limited to) linked to Xlib, they also had libxcb-xlib listed in their .la files dependencies. Which causes everything to fail linking with libtool as it’s looking for the missing libxcb-xlib.la file.

I suppose it’s time to spend time to get a script to fix this situation, but I admit I’m not much motivated at the moment. Especially since my system is pretty slow when it comes to rebuild stuff for testing, and my employer is not going to pay me anytime soon to allow me getting a newer box.

Once the script is available, it should probably be much much easier to get rid of .la files in ebuilds, as we could just say to the users to run the fixing script and be done with that..

But I admit I was planning on doing some different things in the next days, I had little time for myself lately to begin with, and I’m following way too many things at once. Sigh.

A few risks I see related to the new portage 2.2 preserve-libs behaviour

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Mon, 30 Jun 2008 16:48:00 +0000

Fellow developer Zhang Le wrote about the new preserve-libs feature from Portage 2.2 that removes the need for revdep-rebuild.

As I wrote on gentoo-dev mailing list when Marius asked for comments, there are a few prolems with its implementation as it is, in my not-so-humble opinion. (Not-so-humble because I know exactly what I’m talking about, and I know it’s a problem).

Let’s take a common scenario, a system where --as-needed as never used, that is updating a common library from ABI 0 to ABI 1 (so with a change of soname). This library might be, for instance, libexpat.

I don’t want to discuss here what an ABI is and what an ABI bump consists of. Let’s just say that when you make an ABI bump you either remove functions, or you change the meaning of some functions (like the parameters, the behaviour or other things like those).

In the first case the bump is annoying but not much of a problem, executables stop being loaded because symbols are undefined; with lazy-loaded executables, they might die in the middle at the moment the undefined symbols is called, but that’s not our concern here.

The problem comes when a function with the same exact name changes meaning, parameters or return type. In this case, the executable might pass too much or too little data to the function, he pointers might be referring to something completely different, or might be truncated. In general, when you change the interface or the meaning of a function, if the executables built to use the previous version are executed with the new version, they’ll either crash down or behave in a corrupted manner. Which are two subtle issues which we should be looking forward to, as they are hard to debug unless you know about them.

So let’s return to our library changing ABI. Let’s say we have libfooA.so.0 and libfooA.so.1 installed, the first is preserved by preserved-libs, the second is the new one. libfooB.so.$anything links to libfooA as it uses it directly, so it will be in the set of packages to rebuild.

Introducing libfooC.so.$anything that links to libfooB.so.$anything, but as destiny wishes, is also using libfooA.

At this point before the ABI bump we have libfooB depending on libfooA.0, and libfooC depending on libfooB and libfooA.0; after the bump, we decide to rebuild only libfooB, which means that libfooB now depends on libfooA.1 while libfooC is still depending on libfooA.0.

What this means is that, minus symbol versioning, the same symbol would have two (probably different) definitions, which will collide one with the other, leading to subtle crashes, misbehaviour and other fun-to-debug problems.

The problem is that the two ABIs of the libraries are both being loaded in the same userspace, which is a very bad thing, unless the symbols are versioned. On the other hand, symbol versioning is a bit of a mess, it’s not implemented by all operating systems, and I find it quite convoluted.

At the moment I don’t see anything in portage that stops you from shooting in your own foot by doing a partial rebuild. I hope I’m mistaken, but if I’m not, please remember to always do a full rebuild, rather than a partial one. Instead of having programs not starting, you might have programs corrupting your data, otherwise.

Bricolage Break

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Thu, 26 Jun 2008 20:15:00 +0000

I’m taking a break. Enterprise has been turned off for two days by now, I actually spent the whole of today laying in bed with a fan pointed at me, watching National Geographic and Discovery Channel.

I was actually planning to spend a week working on a few DIY projects. The first should have been a very simple one, just some walls for a drawer that’s too big to store stuff in it properly. My idea was to take some plywood, cut it in properly sized rectangles, then cut some slots in them so that they plug one into the other.

I was able to cut slots in two pieces, and was working on the third yesterday, when the mini-mandrel I was using started losing power. I thought it was just overheated but today when I resumed working on the slot it did the same trick, and started having a bad burning smell. I opened it to find the cables looking strangely brown. If you ever seen electrical cables burning, you know the colour.

So now I am without a mandrel to finish the job, which is quite annoying actually, as I use that when I have to fit the new plug modules in the place of the old ones (the old ones were smaller, so the tiles, in the kitchen and the bathrooms, weren’t cut exactly around the boxes).

I suppose this is what happens when you rely on cheap-o-tools. I suppose ecology (and probably economy) would be glad if I actually started buying good tools that last rather than stupid tools that break down in less than two years. Indeed the mini toolset I was using I only paid €18, while a similar set from Dremel (Bosch) is available at around €70.

The problem is that I was planning on buying a router (the powertool, not the network appliance) as I was hoping to work, this summer, on building a bookshelf. Having to replace the toolset is bad news.

So my break is going to be less bricolage-oriented than I hoped. I’ll probably read The Dragon Reborn, although I was hoping to read more society and political books (like the ones I have in my wishlist but I guess that the powertools will use the money I was hoping to use for those.

And I still have to find a way to replace Enterprise.

Disk images

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Tue, 24 Jun 2008 00:44:00 +0000

As I wrote before I’m currently converting my video archive to a format compatible with iTunes, AppleTV and PlayStation3. To do so, I’m archiving it directly on my OSX-compatible partitions, so that iTunes can access it. I’m actually thinking about moving it on an NFS-shared XFS partition, but for now it’s just HFS+.

To do so, though, I had to repartition the external 1TB hard disk, to give more space for the iTunes Library partition. For a series of reasons, I had to backup the content of the disk before, and now I’m restoring the data. To do the backup, I used the DMG image format that is native to OSX.

The nice thing of DMG is that it transparently handles encrypted images, compressed images, read-only images and read-write images. The compressed image format, that is what I used, was able to cut in half the space used up in the partition.

Now of course I know DMG is just a glorified raw image format, and that there are a lot of alternatives for Linux, but this made me think a bit about what the average user knows about disk and partition images.

I know lots of people who think that a raw image taken with dd(1) is good enough to be stored and used for every use. I don’t agree with that, but I can understand what the problem is with understanding why a raw image taken with dd is not just “good enough”.

The problem is that a lot of users ignore the fact that in a partition, together with the actual files’ data, there is space occupied by the filesystem metadata, the journal, and of course all the unused space is not always contiguous and it’s not always zeroed out.

Let’s take a common ad easy example that a lot of users who had to use Windows at one time will have no problem understanding: the FAT32 filesystem.

FAT filesystems tend to fragment a lot. Fragmentation does not only mean that files are sparse around the disk, but also that the free space is sparse between the fragments of files. Also, when you delete a file, its content is not removed from the disk, as you can guess if you ever used undelete-like tools to restore files that were deleted.

When you compress with tools like bzip2 or similar a file, the fact that it’s fragmented does not get in the way of the compression: if the file contains the same data repeated over and over it will be compressed quite easily. If the file is fragmented the same does not apply.

The fact that the free space is not zeroed out can cause lots of problems because a perfectly defragmented partition with 40GB of unused space cannot describe the unused space as “40GB of 0s”… unless.

Unless the compression algorithm knows about the filesystem. If, when you take an image of the partition, you use a tool that knows about the format of the partition, it starts to be much more useful. For FAT, that would mean that a tool could just move all the files’ data at the start of the partition, put all the unused space at the end, and consider it empty, zeroed out. The result is no more a 1:1 copy, but it’s probably good enough.

Now of course an alternative is just to use an archiving tool that can actually get all the files’ metadata (attributes, permissions, and so on), then you don’t have to worry about unused space at all. But that assumes you can use custom tools both for creating the image and to restore it. Creating an image of a partition could be quite easy to do with an already set-up complex tool, but there might be the need for the restore part of the code to be as lightweight as possible.

Now, I’m no expert on filesystems myself, so I might be wrong, or there might be the software doing this already out there. I don’t know. But I think it wouldn’t be too far fetched to think that there might be a software capable of taking an hard drive with an MBR partition table, with two FAT32 filesystems on it, both fragmented, with the unused space not zeroed out at all, and creating an image file that only needs bzip2 to be uncompressed, and dd to be written, but still providing a much smaller image file than a simple raw image compressed with bzip2, or maybe even rzip.

Such a tool could easily find duplicated files (which happens a lot on Windows because of duplicated DLLs, but can easily happen on Linux too because of backup copies for instance), and put them one near the other so to improve compression.

I know this post is quite vague on details, and that’s because as I said I’m not much of an expert on the field. I just wanted to make some users reflect on the fact that a simple raw image is just not always the perfect solution if what you need is efficiency in storage space.

rbot and Gentoo

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Mon, 23 Jun 2008 20:29:00 +0000

One nice thing about rbot is certainly, to me, the fact that installing it is mostly just an emerge away. For quite a while there has been a live ebuild of rbot that fetched first from Subversion and now from GIT.

The only problem was that the ebuild fetched the sources, then made a gem, installed it, and then removed some files. Not exactly the shortest way around.

Thanks to the guys in #rbot, this has been solved. Now rbot installs itself without using gems, but by simple setup.rb. This means not only that you won’t need rake and zip to install it, but also that the installed files have decent path, rather than being buried inside the rubygems directories.

An additional advantage comes from the size of the installed files; when you install a gem, a copy of the sources is left in the rubygems tree; this can be quite a waste. The size of the rbot binpkg has halved:

-rw-r--r-- 1 root root 959426 May 24 12:08 /usr/portage-packages/All/rbot-9999-r8.tbz2
-rw-r--r-- 1 root root 425034 Jun 23 20:18 /usr/portage-packages/All/rbot-9999-r9.tbz2

Add to this that the ruby-gettext dependency is gone too, now you got a nls USE flag that allows you to choose whether you want localised bots or not, and if you don’t want them, it won’t ask you for ruby-gettext at all. Nice, uh?

But it’s not finished here. I was hacking at the ebuild today for a different reason. rbot has a figlet plugin, that executes the figlet command and copy the output on IRC. The ebuild, though, didn’t have an option to pull in figlet, and I wanted to fix that.

So together with the removal of rubygems dependency, the new ebuild also has USE flags to enable the figlet, cal, host and fortune plugins. So you either get your dependencies pulled in, or the bot has the plugins disabled out of the ebuild. Cool!

And talking about the fortune plugin, listing the categories didn’t work on Gentoo, as we install the database in a different directory than rbot expected them to be. This is fixed in rbot upstream repository by yours truly.

Always a pleasure to fix rbot on Gentoo ;) Kudos welcome as well as bribes.

I bought a software license

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Sun, 22 Jun 2008 18:11:00 +0000

I finally decided to convert my video library to Mpeg4 containers, H.264 video and AAC audio, rather than mixing and matching what I had before that. This is due to the fact that I hardly use Enterprise to watch video anymore. Not only because my office is tremendously hot during the summer, but more because I have a 32” TV set in my bedroom. Nicer to use.

Attached to that TV set there is an Apple TV (running unmodified software 2.0.2 at the moment) and a PS3. If you add to that all the other hardware that can play video I own, the only common denominator is H.264/AAC in MP4 container. (I also said before that I like the MP4 format more than AVI or Ogg). It might be because I do have a few Apple products (iPod and AppleTV), but also Linux handle this format pretty well, so I don’t feel bad about the choice. Beside, new content I get from youtube (like videos from Beppe Grillo’s blog) are also in this format—you can get them with youtube-dl -b.

Unfortunately, as I discussed before with Joshua, and as I tried last year before the hospital already, converting video to this format with Linux is a bit of a mess. While mencoder has very good results for the audio/video stream conversions, producing a good MP4 container is a big issue. I tried fixing a few corner cases in FFmpeg before, but it’s a real mess to produce a file that QuickTime (thus iTunes, and thus the Apple TV) can accept.

After spending a couple more days on the issue I decided my time is worth more than what I’ve been doing, and finally gave up to buy a tool that I have been told does the job, VisualHub for OSX. It was less than €20, and that is usually what I’m paid by the hour for my boring jobs.

I got the software, tried it out, the result was nice. Video and audio quality on par with mencoder’s but a properly working MP4 container that QuickTime, iTunes, AppleTV, iPod and even more importantly xine can play nicely. But the log showed a reference to “libavutil”, which is FFmpeg. Did I just pay for Free Software?

I looked at the Bundle, it includes a COPYING.txt file which is, as you might have already suspected, the text of GPL version 2. Okay, so there is free software in here indeed. And I can see a lot of well-known command line utilities: lsdvd, mkisofs, and so on. One nice thing to see is, though, an FFmpeg SVN diff. A little hidden, but it’s there. Good.

The doubt then was if they were hiding the stuff or if it was shown and I did just miss it. Plus it has to have the sources of everything, not just a diff of FFmpeg’s. And indeed in the last page of the documentation provided there is a link to this that contains all the sources of the Free software used. Which is actually quite a lot. They didn’t limit themselves to take the software as it is though, I see at least some patches to taglib that I’d very much like to take a look to later—I’m not sharing confidential registered-users-only information by the way, the documentation is present in the downloadable package that acts as a demo too.

I thought about this a bit. They took a lot of Free Software, adapted it, written a frontend and sold licenses for it. Do I have a problem with this? My conclusion is that I don’t. While I would have preferred is they made it more clear on the webpage that they are selling a Free Software-based package, and that they would have made the frontend Free Software too, I think they are not doing anything evil with this. They are playing well by the rules, and they are providing a working software.

They are not trying to exploit Free Software without giving anything back (the sources are there) and they did something more than just package Free Software together, they tested and prepared presets to use for encoding for various targets, included Apple TV which is my main target. They are, to an extent, selling a service (their testing and presets choices), and their license is also quite acceptable to me (it’s like a family license, usable on all the household’s computers as well as a work computer in an eventual office).

At the end of the day, I’m happy of spending this money as I suppose it’s also going to further develop the Free Software part of the software too, although I would have been happier to chip in a bit more if it was fully Free Software.

And most importantly, it worked out of the tarball solving me a problem I was having for more than an year now. Which means, for me, a lot less time spent trying to get the whole thing working. Of course if one day I could just do everything with simply FFmpeg I’ll be very happy, and I’ll dedicate myself a bit more on MP4 container support, both in writing and parsing, in the future, but at least now I can just feed it the stuff I need converted and dedicate my time and energy toward more useful goals (for me, as in paid jobs, and for the users with Gentoo).

Debuking ccache myths

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Sat, 21 Jun 2008 20:22:00 +0000

Doing support work in #gentoo-it is probably one of the most user-facing tasks I’ve been doing lately, it’s nice because you can often gather common misconceptions about problems and tools.

One of these is related to ccache. Some users seem to think that ccache will improve their compile speed in any situation. This couldn’t be more wrong.

First of all, in the situation of an always different source file, ccache will make build take a longer time than a non-cached build. The reason is pretty simple once you think of it. The cache is indexed by an hash, the md5 of the preprocessed source file; and the content of the cache is the resulting object file. When you build a given source file, ccache will have to take an md5 hash of the preprocessor’s result. Then it should look for it on the cache tree, and if it’s not found, it will have to compile it and write the output twice (once as the output of the build and once in the cache). It might not be a huge overhead but it’s an overhead nonetheless.

So there has to be a benefit to use ccache for it to be useful. The benefit is that, when you build the same sources twice, the hash, lookup and copy takes less time than the build, usually. But when do you actually build the same sources twice?

The first myth to debunk is that it’s helpful for packages using libtool as they build sources twice (one PIC and one non-PIC). While it’s true they build the same sources twice, they are not compiled in the same way, so the cache is not saving anything. If they were built the same way, there would be no reason to actually build it twice, no?

The second is that ccache helps when you change your CFLAGS. The idea of ccache is that it gives you the exact output the compiler would give you. And this means that changing CFLAGS will change the resulting output too. If it was ignoring the change in CFLAGS and returning data from cache it would be breaking your setup by disallowing you to change CFLAGS. Again, ccache is not helping you.

The third myth is that ccache makes changing USE flags a matter of seconds rather than hours. While it’s true that this is a case where commonly you do have an advantage on using ccache, it’s not that simple. Changing USE flags usually means changing the compiled code; there are rare cases (like xorg-server PDEPENDs) that allows you to keep the same exact sources when changing USE flags.

Even then, if you change versions of the libraries used by the software, then the preprocessed sources will change, and we’re back to square one.

All in all, ccache is not bad, it’s helpful in various situations, it’s quite useful for developers. But it’s not a panacea for Gentoo users.

My take on Genji: Days of The Blade

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Thu, 19 Jun 2008 19:08:00 +0000

So, I named Genji before, and I said it’s a cool game. Well, I have some details to fix up on that statement.

The game in itself looks tremendously cool. Very nice graphics, nice characters, nice story. But it has some pitfalls in the actual implementation.

First of all I have to say that the graphics is not a way to get the eyes off from the actual gameplay. The gameplay is nice. Of the four playable characters, none is the perfect character: this means that you have to switch between them to balance the game. This is very good to me, otherwise I end up fixating myself on one playable character and use just that. The fact that the mini-bosses need to be hit on the back otherwise they just won’t come down is a very nice touch in gameplay. And in general the whole game is well designed.

The problem comes on levels design. I’m in what is probably the third “area” (not third level), the checkpoints where you recharge and save are very distant one with the other, which isn’t bad on its own, but it starts to be a problem when you don’t provide more than a couple of health-restoring items, and you start having something like six mini-bosses to fight (with their team of minions) before you can get near a new checkpoint.

Sometimes it’s just a waste of time because you have to get back to the previous checkpoint to save and recharge your health. Sometimes it’s simply a pointless exercise because you end up fighting more and more minions, as they re-appear every time you come around the same corner, even though you fought and killed them all.

Now this is not entirely bad, as you can’t just turn back and pass again a level to become stronger before challenging a boss. Which is quite annoying when you have a savegame in the boss’s lair, and you come to think your characters are not strong enough to cope with that.

So what do I think of the game, all things considered? A bit frustrating. Not only because of the distance of checkpoints, but even more because it is tremendously well designed in other areas that this frustration is not easy to explain.

I’m not regretting spending money to get this game though, of that I can be sure.

I guess my next phone is not going to be a Nokia

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Wed, 18 Jun 2008 21:20:00 +0000

I wrote a few times before about my experience with my Nokia E61 phone. WIth the iPhone 3G coming to Italy, and the E71 being announced, I started to think whether to change my phone, and which one to get.

The first issue would be, do I want to keep going with Nokia or not? Nokia is being felt as “great’ in the Free Software world, although there has been some trouble with Ari Jaaksi’s words lately but I’m very much disliking their policies related to phones.

The first problem is something I wrote before about, and it’s their SDK policies. Not only the S60v3 SDK is not usable under Linux, but also the most complete SDKs are just overpriced.

Yes, I know Apple’s SDK isn’t much better, but this is just one issue I have with Nokia.

The other big issue with Nokia is that even the most basic annoyances you might have with the phone’s software will never get fixed in your phone’s software release. For instance, E61 has a pretty useless home screen, it shows you the next appointments in the calendar and a single TODO item. With the huge screen real estate in the phone, it’s kinda pointless to have just these things in there.

The problem has being partly solved in the E61i version, which has the same exact screen size, but has a more interesting home screen, which includes the number of new unread email messages, and the presence of Wireless LAN connections. I don’t think that to implement the new home screen you need any new hardware the E61i has and the E61 doesn’t. But Nokia is not going to update the firmware of E61 with that, as it’s a service release of the OS.

The problem is that there are tons of little annoyances in Nokia’s Symbian, but to get those fixed your only choice is for you to buy a new phone. Which would be just a waste of money. They really should start updating the OS for old models too, not just for the last version they released of every model.

Now, as I was saying the other day talking with DanielW on the Amarok developers’ channel, unfortunately the most open platform currently in use and with a decent price tag is Windows Mobile. Actually, Windows Mobile phones do look nice, but I need to sync with OSX at the moment (Address Book beats Kontact a million times, for now, once KDE gets a decent address book application, then I’ll have KDE as main sync I suppose), and I heard so many bad things about Windows Mobile (and one I seen with my eyes, my brother-in-law was receiving a photo of my nephew with his Windows Mobile PDA via bluetooth, it ended up stuck, he tried shutting it off, it didn’t work, he removed the battery and put it back, and it reset itself entirely…).

Sadly, there isn’t so much choice here. I want a smartphone, as I grew used to have email support while I’m on the go, for job and other things; also, being able to browse some e-shops when I’m at an actual shop, to check for prices did save me quite a bit of money before. There are only three choices for smartphones out there: Symbian OS, Windows Mobile and iPhone. The first, as I said, is a huge pain in the ass as it is almost never updated; the second is just not something I’d be glad to be using, so at the moment my best choice is the third.

Now of course before I can actually get an iPhone there are a few things I need to clear. First TIM has to announce it. I’ll never become a Vodafone customer in my life, they are just thieves. And TIM has to announce a decent dataplan as I need it. As an alternative they can just not simlock it, I’ll be glad to pay for a decent voice plan to give my mother and keep the iPhone with my current carrier (H3G) which has a very nice dataplan.

But for sure, the only way for my next phone to be Nokia would be for me to work for Nokia and having the tools to actually update its firmware to fix the bloody annoyances it gives me, like not being able to search for contacts by nickname when looking up the destination of an SMS, or showing the birthdays in the calendar view (and the homescreen) for those contacts having one set.

Why conditional patching and autotools rebuild is a Bad Idea

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Sat, 14 Jun 2008 13:00:00 +0000

With capital B and I, of course.

I wrote yesterday about maintainer mode use by ebuilds and why it’s a problem. Today I want to focus on a slightly related issue: conditional patching and autotools rebuild.

One very common use of maintainer mode is due to conditional patching: you can’t inherit autotools.eclass conditionally, so you leave it up to the autotools to decide whether to rebuild themselves or not.

This is bad not only for the problems I outlined for maintainer mode itself, but also because conditional patch, as well as conditional autotools rebuild, is something you should not do in an ebuild unless there is an extremely good reason to.

The first problem is quite obviously that you cannot send conditional patches upstream (see also my old article Best practices for portable patches). In Gentoo we should really prefer patches taken from upstream and/or that can be sent and accepted by upstream.

But it’s not limited to that: conditional patches gets applied only on particular conditions, this means that if you are bumping the package and those conditions don’t apply to you, then you won’t be seeing whether the patch applies or not. And if it’s an important fix for a particular architecture, and the patch cannot apply, then that architecture will have a broken package. If it’s a build-fix you can live with it a bit (although it will upset with reason the users of that arch), but if it’s a runtime fix then it’s a problem, because it might go months without being noticed.

And it’s not just during bump, but also when you add a new patch, it might not stack correctly against one of the conditional patches, and you wouldn’t be seeing the failure. So for instance to fix a build error on one setup you might break build on a different one.

In general, you want the patches to be unconditional, and you want to rebuild autotools at the minimal change. This even if the rebuild would be needed by a reduced set of users. The reason is that it’s better to waste a little more time for all the users rather than having a (big or small) subset of them being stuck entirely because of a conditional patch or rebuild.

Also, the sooner you can send a patch upstream, the sooner you won’t be needing it in the ebuild. If you prepare a conditional, raw and quick patch to fix a setup, you cannot send it as it is to upstream, so you’ll have to either spend more time to prepare also a refined good patch, or you’ll have to deal with that patch for a long time. Neither solution is a good one for your workflow.

So please, don’t make anything conditional, unless it’s really really needed.

Maintainer mode

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Fri, 13 Jun 2008 14:12:00 +0000

Developers happen to edit Makefile.am and configure.ac a lot during the development of any software. For this reason, a long time ago, autotools included a support called maintainer mode.

When maintainer mode is active, at a change to those files, as well as other autotools files, the proper tool is called (autoconf, automake, etc) to rebuild the whole series of files. This is useful because a simple make call does almost everything for the developer.

Recently, though, maintainer mode is no more optional by default, but it’s instead force-fed on every project, unless AM_MAINTAINER_MODE is explicitly called in the configure.ac file.

This is all well and good, so why am I talking about this? Well because maintainer mode is bad to use in ebuilds. If you patch a Makefile.am or configure file, and you don’t explicitly rebuild autotools, maintainer mode will do it for you, but it causes a few problems this way.

The most annoying problem for users is that if the maintainer mode is triggered by an edit to configure.ac, the build will run ./configure twice. It’s boring enough to run once.

But more problems arise for instance when a new version of autoconf or automake is released. Because in that case the versions will not be the same between the original and the one that would be built, aborting the whole build (see my autofailure guide that reports that for instance).

Also, if you leave the task up to maintainer mode, it won’t even take into consideration the WANT_AUTO* variables that we use to force a particular version of autoconf and automake, so you’ll have automagic dependencies. And most likely you’ll forget to add autotools to the dependencies of the ebuild.

Okay so we know why maintainer mode is bad, how do we make sure that ebuilds don’t trigger maintainer mode? Beside the fact that developers should know what they do, by rebuilding autotools explicitly when patching the files, it’s worth looking “postmortem” if an ebuild triggered the maintainer mode rebuild. It’s actually useful because sometimes upstream is crazy and fixes the tarball without going through make dist and maintainer mode is triggered for unpatched vanilla tarballs too (in that case you should rebuild autotools explicitly anyway).

To check if a build triggered maintainer mode, you can just grep the build log for the string “missing --run” (without quotes of course), that is the announce that maintainer mode was triggered. If that’s the case… it’s time to file a bug to latch to this tracker .

I suppose I’ll have to fix bugs on a few more packages I don’t use/care about myself directly in the next weeks. Sigh, I need the new box.

Please don't build your examples by default

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Wed, 11 Jun 2008 16:45:00 +0000

I’m in the middle of an emerge -e world I started to try getting as much glibc 2.8 failure have a bug already on Gentoo’s Bugzilla so that users hitting them don’t have to report them anew.

One thing I noticed during this rebuild is the amount of time spent with no good reason to build tests and examples.

I have the test feature disabled, I only enable it for the software I maintain, and I usually don’t have so much time to look at the test of all software. Still some packages, like dbus, hal, gtk, and so on, build their tests anyway. They don’t run them, but they build tests and example during the default make all call.

I talked with Lennart about that some time ago as also libdaemon does that. He pointed out to me that these examples are often a way for the developers to test that the code still links correctly, for instance, and thus should not be disabled by default during developmet. I agree it can be useful that way.

For this reason, I prepared a patch for libdaemon, that you can now find here, which I should have committed and I forgot about – I’ll see to do that soonish :P – which should make it possible to have the best of both options: developers still get their examples (and/or tests), while distributions can opt-out them.

For ebuilds, this means that the src_test function should do a make -C tests or make -C examples to build tests and examples, after such a patch is applied, to explicitly build the code that most users won’t need during standard run.

I know it’s just seconds, or minutes, of time to build the examples, but if you add all those up, you’re wasting lots of time. And we should always strive to optimise time usage, no?

Sub-optimal optimisations?

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Wed, 11 Jun 2008 14:41:00 +0000

While writing Implications of pure and constant functions I’ve been testing some code that I was expecting to be optimised by GCC. I was surprised to find a lot of my testcases were not optimised at all.

I’m sincerely not sure whether these are due to errors on GCC, to me expecting the compiler to be smarter than it can feasibly be right now, or to the “optimised” code to be more expensive than the code that is actually being generated.

Take for instance this code:

int somepurefunction(char *str, int n)
  __attribute__((pure));

#define NUMTYPE1 12
#define NUMTYPE2 15
#define NUMTYPE3 12

int testfunction(char *param, int type) {
  switch(type) {
  case 1:
    return somepurefunction(param, NUMTYPE1);
  case 2:
    return somepurefunction(param, NUMTYPE2);
  case 3:
    return somepurefunction(param, NUMTYPE3);
  }

  return -1;
}

I was expecting in this case the compiler to identify cases 1 and 3 as identical (by coincidence) and then merge them in a single branch. This would have made debugging quite hard actually (as you wouldn’t be able to discern the two case) but it’s a nice reduction on code, I think. Neither on x86_64 nor on Blackfin, neither 4.2 nor 4.3 actually merge the two cases leaving the double code in there.

Another piece of code that wasn’t optimised as I was expecting it to be is this:

unsigned long my_strlen(const char *str)
  __attribute__((pure));
char *strlcpy(char *dst, const char *str, unsigned long len);

char title[20];
#define TITLE_CODE 1
char artist[20];
#define ARTIST_CODE 2

#define MIN(a, b) ( a < b ? a : b )

static void set_title(const char *str) {
  strlcpy(title, str, MIN(sizeof(title), my_strlen(str)));
}

static void set_artist(const char *str) {
  strlcpy(artist, str, MIN(sizeof(artist), my_strlen(str)));
}

int set_metadata(const char *str, int code) {
  switch(code) {
  case TITLE_CODE:
    set_title(str);
    break;
  case ARTIST_CODE:
    set_artist(str);
    break;
  default:
    return -1;
  }

  return 0;
}

I was expecting here a single call to my_strlen(), as it’s a pure function, and in both branches it’s the first call. I know it’s probably complex code once unrolled, but still gcc at least was better at this than intel’s and sun’s compilers!

Both Intel’s and Sun’s, even at -O3 level, emit four calls to my_strlen(), as they can’t even optimise the ternary operation! Actually, Sun’s compiler comes last for optimisation, as it doesn’t even inline set_title() and set_artist().

Now, I haven’t tried IBM’s PowerPC compiler as I don’t have a PowerPC box to develop on anymore (although I would think a bit about the YDL PowerStation, given enough job income in the next months—and given Gentoo being able to run on it), so I can’t say anything about that, but for these smaller cases, I think GCC is beating other proprietary compilers under Linux.

I could check Microsoft’s and ~~Borland’s~~ Codegear’s compilers, but it was a bit out of my particular scope at the moment.

If I did think a bit before about supporting non-GNU compilers for stuff like xine and unieject, I start to think it’s not really worth the time spent on that at all, if this is the result of their compilations…

Recognising glibc 2.8 failures

flameeyes@gmail.com (Diego "Flameeyes" Pettenò) — Mon, 09 Jun 2008 13:06:00 +0000

Mike already wrote to gentoo-dev about the new build failures with glibc 2.8. Thanks GNU, really, for starting the long overdue cleanup.

I say long overdue because other C libraries like FreeBSD’s and others already had headers that included the very minimum needed. So some of the issues that we’re going to fix now were just as valid for FreeBSD and others.

So this is going to help portability to FreeBSD and viceversa, stuff already ported to FreeBSD should build just fine with the new glibc version.

As I’m borderline masochist, I started looking at patches to see if I can easily tell if they are good and commit them myself as needed (yeah I know I step over lots of toes; with most people I got agreements before though, so sometimes it’s just fine, beside as long as the patch works and I don’t break stuff nobody complained recently ;) ).

One thing I noticed was a misfiled bug, it reported failure with glibc 2.8 when it was instead a failure with gcc 4.3. The issue is more or less the same, missing includes most of the time, because both projects cleaned up their header files, but it’s important to know which package caused it to make sure that the faulty ebuild are all stable before gcc and glibc can make stable.

For the list of bugs caused by glibc 2.8 I refer you to the tracker as you can see there, the usual problems are:

PATH_MAX undeclared;
ARG_MAX undeclared;
field ‘reg’ has incomplete type;
storage size of ‘peercred’ isn’t known.

The solution for some of these issues is to add the right include (like limits.h), for others is to properly define _GNU_SOURCE to get the GNU extensions enabled. Usually projects using autotools already have a check for that, but recent autoconf versions use config.h to define that, and not all the sources include it.

On ebuilds you can work that around by using append-flags -D_GNU_SOURCE but upstream should fix their source files by including config.h right at the start of every source file.

For failures related to C++ code, like undeclared ostream or cerr, the problem is not with glibc 2.8 but with gcc 4.3 instead.

Enjoy!