Flameeyes's Weblog

Random notes about xine plugins interface

Posted by Diego "Flameeyes" Pettenò Wed, 30 Jan 2008 23:40:00 GMT

Here it comes a new technical blog, and this time it’s also related to xine,s o you get a double important post.

xine, like many other modern software, uses a plug-in architecture. This is pretty nice especially for a multimedia application, as it allows even binary distributions to provide granularity like Gentoo does with the USE flags, and allows the processes not to load all the backend libraries that aren’t needed at once.

Or does it?

Unfortunately the architecture of xine has a few problems. One of these is that at the moment, almost every plugin is loaded at application startup, rather than as needed. The first time it’s comprehensible, as xine need to produce a cache file, but the other times?

The first problem was with settings: a lot of plugins had settings to be loaded at startup, but luckily Thibaut took care of this and implemented cache for settings too. This solves one problem.

Then there is another problem: most of the times we can’t trust the extension of a file, so the default is to test the demuxer by content. Testing the demuxer by content requires loading all the demuxers and try them one by one. This is expensive. The reverse detection method is good for files, as we first try to identify the demuxer by the extension, if that doesn’t work, and just then, we do load all the demuxers and try them one by one. For stream, we can’t trust the extension at all.

So what are the problems here? Well first off even if we decided to use the reverse method as default, we’ll be losing all the advantage at the first file that fails the extension detection or at the first stream; then there is the problem that we end up even testing demuxers that, well, can’t just play files. This is the case of the cdda demuxer, for instance, or some network demuxers.

What we need then is to solve two problems:

detect streams without using the by-content detection, this can be done once I’ll have time to complete reading mime-types from input (HTTP Content-type, but also Content-type saved in extended attributes for local files); this should solve the problem of running content-detection for every stream; most stream sources will report correct Content-type, as most likely would gnome-vfs and other VFS implementations;
get a way to tell xine-lib that a given demuxer works only with a certain input plugin (and vice-versa, as the dvd demuxer will always use the same mpeg demuxer).

This solves a bit of problems by itself, by avoiding loading unneeded plugins, but the architecture of xine’s plugin system is still rough.

That happens because of the way the data is currently exported by the plugin itself, through a global exported structure, containing the a literal string, and the pointer to a static function used to allocate and fill the next structure. This structure contains pointers, the pointers need to be relocated, so it’s in .data.rel.ro, which gets COW’d once the plugin is loaded and relocated; as prelink does not work with plugins, we can’t even rely on that to solve the problem for us.

With the COW, all the xine plugins gets 4KB of memory used up per plugin, per process.

What I would like instead would be a new xine_plugin_get_info() which allocates in heap the structure, and fills it. It will be slower most likely, but it will a) avoid the difference between structures loaded from cache (which are allocated in heap), and those loaded from plugins (which are mapped in .data.rel.ro) b) avoid the 4KB waste per plugin per process, by just allocating the right amount of memory needed to keep the structure in memory.

If I have extra time I might actually try this on a branch, it might be a decent improvement after all.

Posted in English | Tags C, COW, plugins, Tech, xine | no comments

Another power outage

Posted by Diego "Flameeyes" Pettenò Wed, 30 Jan 2008 14:21:00 GMT

It seems to be a common factor every month, the power company disconnects us for a few hours (between 3 and 6, every time longer than my two UPSes can sustain) and I’m forced to use the UMTS phone to connect.

Today I was also going to wrk all day long if it wasn’t for this not-so-small problem.

Anyway, I put a few more photos on the Sony Reader photoset . At the end I bought the Postfix book from O’Reilly, as I had to fix a thing in xine-project’s bugzilla already—and now I might be able to learn how postfix works for once, and I used it for the photo. Later I’ll prepare a photo with the Not So Short Guide to LaTeX2e optimised for the Reader as I promised.

Now it would be nice if O’Reilly started providing Reader-sized PDFs in the future ;)

Posted in English | Tags ENEL, Photos, Power, PRS505, Sony, SonyReader | no comments

My impressions about the Reader

Posted by Diego "Flameeyes" Pettenò Mon, 28 Jan 2008 22:42:00 GMT

So I finally received the Sony reader I ordered almost a month ago. Actually the shipment was quite fast, sent on January 20, received today.

First impressions with the hardware are positive, it’s a bit more heavy that I would have thought, but it’s big enough, and the eInk display is really good. Quite a nice item. PRS-505 photos here

An half problem is having the software working. There is libprs500 which takes care of almost everything, but packaging that is becoming a bit of a challenge.

While the author is a Gentoo user, and very helpful to improve the situation, I found a bit of problems now with xdg-utils. The post-installation script of libprs500 uses xdg-utils commands to install icons, desktop files and similar stuff. Unfortunately xdg-utils is … far from perfect. 81 open bugs on FreeDesktop’s bugzilla, and a lot of gray areas.

First off, xdg-utils don’t support DESTDIR (nor does the postinst script, but that I fixed); this means that it tries to write directly on filesystem, which is not good at all for distributions, not only Gentoo. I can workaround some of these problems by setting XDG_DATA_DIRS to a modified path forcing it to use the correct DESTDIR.

Even worse, xdg-mime and xdg-desktop-menu don’t even use XDG_* variables, they install data for GNOME and KDE separately, and for KDE, they use, respectively, kde-config ’s output, and nothing, just hardcoding the path. I was able to fool xdg-mime to work as I need by faking a kde-config script, but for xdg-desktop-menu there is nothing I can do. Beside the ability to use DESTDIR, I could have fooled them enough if they at least used KDEDIR/KDEDIRS variables, as I suppose they should, but they don’t.

Hopefully I’ll be able to get a modified xdg-utils soon so that I can actually complete the ebuild for libprs500, and then add it to portage.

I’m still having one problem with the connection of the PRS-505: the SD card in the slot is not seen by Linux. It’s working, because I can see it working on OSX, but on Linux somehow it does not appear to be given a device at all. I suppose it should have a device assigned like a 50-in-1 flash card reader, but this does not currently seem to happen, and I don’t know yet why.

Beside that, libprs500 is a nice frontend, it’s complete, not rough at all, and quite appealing. The only problem with the software itself I have is that it uses a sqlite db to store the books. No, not the books metadata, but the books themselves, it saves the whole file into the database. As you can guess, this is far from optimal, considering also my pet peeves with sqlite, I’d very much like to try steering upstream to something different, especially because I want to load something like 300 MB of books on the Reader.

As for what concern the kind of books to load on it; A4 books work nicely when used landscape and zooming in; tomorrow night I’ll experiment a bit with paper types for texinfo manuals, so that I could generate GDB, Make and ELISP manuals in a suitable size for the reader itself. The conversion to the reader’s own format is not that good when you have complex PDFs from texinfo or LaTeX.

There is one non-small problem with O’Reilly’s openbooks, like LDD3: the PDF has the guides printed around the page, and the zoom function of the reader is thrown off by that (it removes the white borders, but then the white border is interrupted by the guide on those books. To read those easily on the reader, the trick would be to crop them; I should look into tools to handle that, there has to be something there able to do that.

For what I’ve seen up to now, it was worth buying.

Posted in English | Tags Books, OReilly, PDF, PRS500, PRS505, Reader, Sony | 3 comments

Some notes about multi-threading

Posted by Diego "Flameeyes" Pettenò Sat, 26 Jan 2008 16:43:00 GMT

Ah, multithreading, such a wonderful concept, as it allows you to create programs in such a way that you don’t have to implement an event loop (for the most part). Anybody who ever programmed in the days of DOS has at least once implemented an event loop to write an interactive program. During my high school days I wrote a quite complex one for a game I designed for a class. Too bad I lost it :(

Multithreading is also an interesting concept nowadays that all the major CPUs are multi-core; for servers they were already for some time, but we know that mainstream is always behind on these things, right?

So, now that multithreading is even more intersting, it’s important to design programs, but even more importantly, libraries to be properly multithreaded.

I admit I’m not such a big expert of multithreading problems, I admit that, but I do know a few things that come useful when developing. One of these is that static variables are evil.

The reason why static variables are evil is because they are implicitly shared between different threads. For constants this is good, as they use less memory, but for variables this is bad, because you might overwrite the data another thread is using.

For this reason, one of the easiest thing to spot in a library to tell if it’s multithread-safe or not is to check if it relies on static variables. If it does, it’s likely not thread safe, and almost surely not thread optimised.

You could actually be quite thread safe even when using static variables, the easy way to do that is to have a mutex protecting every and all accesses to the variable, this way only one thread at a time can access it, and noone can overwrite someone else’s data.

That cause a problem though, as this serialises the execution of a given function. Let’s take for instance a function that requests data through the net with an arbitrary protocol (we don’t care which protocol it is), saves it on a static buffer, and then parse it filling a structure with the data received and parsed. If such a function is used in a multithreaded program, it has to be protected by a mutex, as it uses a static buffer. If four threads require access to that function almost simultaneously (and that might happen, especially on multi-core systems!), then the first one arriving will get the mutex, the other three will wait till the first one completed processing. Although in general, on a multicore system you’d then have other processes scheduled to be executed at that point, you’re going to waste time by waiting for a thread to complete its operation, before the next one can be resumed.

This is extremely annoying, especially now that the future of CPUs seems to be an increase in number of cores, rather than in their speed (as we’re walking around a physical limit of 3GHz as far as I can see). The correct way to handle such a situation is not to use a static buffer, but rather use a heap-allocated buffer, even if that is slightly slower for a single thread (as you have to allocate the memory and free it afterward); this way the four threads are independent and can be executed simultaneously. For this reason, libraries should try to never use static buffers, as they might not know if the software using them is multi-threaded or not.

When a library is blatantly not thread-safe, there is even a bigger problem, which can be solved in two ways: the first is to limit access to that library to a single thread. This way there are no problems with threading, but then all the requests that need to be sent to that library has to be passed to the thread, and the thread has to answer to them; while cheaper than IPC, ITC is still more expensive than using a properly thread-safe library.

The other option is to protect every use of the library with a mutex. This makes a library thread-safe if it’s at least re-entrant (that is, no function depends on the status of global variables set by other functions), but acts in the same way as the “big kernel lock” does: it does not allow you to run the same function from two threads at once, or even any function of that library from two threads at once – if the functions use shared global variables.

How should libraries behave, then, when they need to keep track of the state? Well there easiest way is obviously to have a “context” parameter, pointer to a structure that keeps all the needed state data, allowing two threads to use different contexts, and call the library simultaneously.

Sometimes, though, you just need to keep something similar to an errno variable, that is global and set by all your functions. There’s no way to handle that case gracefully through mutexes, but there’s an easy way to do that through Thread-Local Storage. If you mark the variable as thread-local, then every thread will see just one copy of that variable, and doesn’t need an explicit mutex to handle that (the implementation might use a mutex, I don’t really know the underlying details).

This is also quite useful for multi-threaded programs that would like to use global variables rather than having to pass a thread structure to all the functions. Take this code for instance:

/* Instantiated a few times simultaneously */
void *mythread(void *address) {
  mythread_state_t *state = malloc(sizeof(mythread_state_t));

  set_address(state, address);
  do_connection(state);
  check_data(state);

  do_more(state);
}

While for library API calls having a context parameter is an absolutely good idea, if the code has no reason to be reentrant, passing it as parameter might be a performance hit. At the same time, while using global variables in libraries is a very bad idea, for programs it’s not always that bad, and it can actually be useful to avoid passing parameters around or using up more memory. You could then have the same code done this way:

__thread mythread_state_t thread_state;

/* Instantiated a few times simultaneously */
void *mythread(void *address) {
  set_address(address);
  do_connection();
  check_data();

  do_more();
}

The thread_state variable would be one per thread, needing neither a mutex to protect it, nor to e passed once to every function.

There are a few notes about libraries and thread safety which I’d like to discuss, but I’ll leave those for another time. Two tech posts a day is quite a lot already, and I need to resume my paid job now.

Posted in English | Tags C, Programming, Threads, TLS | 5 comments

A good kind of cow

Posted by Diego "Flameeyes" Pettenò Sat, 26 Jan 2008 14:36:00 GMT

After all I wrote before, I’m sure at least some people might think that every COW is a bad COW, and that you should never use COW sections in your programs and libraries.

It’s not exactly this way. There are times when using a copy on write section like .bss is a good choice over the alternatives.

This is true for instance for buffers: there are mainly three ways to handle buffers: malloc() allocated buffers, automatic buffers allocated on the stack, and static buffers that are added to .bss.

A buffer allocated on the stack has the main advantage of not having to be explicitly free()‘d, but big buffers on the stack, especially if not well warded, might cause security issues. Plus it might require a big stack.

A buffer allocated in heap through malloc() is more flexible, as you only request memory as needed, and free it as soon as it’s not needed anymore (for stack-based buffers, you need to wait the end of the block, or create a block for the instructions to be executed that use the buffer). This reduces the memory footprint when looked at in time, but it has a little overhead as malloc() and free() are called.

Another option is to use static arrays as buffers. Non-initialised static arrays are put into .bss which is a copy on write section that is usually backed against the zero page (although I’m not yet sure how the changes in Linux 2.6.24 about zero page affect this statement). The good thing about having static buffers is that you don’t need to manage their lifetime, neither explicitly, nor implicitly, as they are already allocated at the start of the program.

This is not good for libraries, as you might have a static buffer in .bss which is never used, but still takes up memory once copy on write of other, used, .bss variables are modified. The thing is better for simple fire-off programs, which starts and terminate quickly (non-interactive programs).

It’s also important to note that libraries should always be designed for work in multi-threaded software as a good design principle, and that static variables and arrays there are not much useful, unless they are all marshalled by the presence of a mutex (which will reduce performance). For this reason, .bss is a bad thing, for libraries, in almost all cases.

For fire-off programs, as I said this is less of a problem, as the buffer might just be used a few couple of times during the life of the program, and if it’s reasonably sized, it might not even impact the whole memory usage of the program (even a single static variable, once changed, will require you to waste a 4KiB memory page, so if you add a 100 bytes variable, that will not change; it will change if you use a 4KiB, or bigger, static buffer).

So sometimes you just have to give up, the static buffer might be increasing the performance of the program, so it just has to stay there. This is why I don’t really fight with .bss too much, the only thing that I don’t think should ever go to .bss are tables: calculating them at runtime is useful only for single task embedded systems, so there has to be a way to opt out from that by using hardcoded tables calculated before or right at build time.

Another good use of .bss is when the memory would just be allocated at the start of the program and then freed at the exit. This is often the case in xine-ui for instance, as there are big structures with all the state information for a certain feature. Those data cannot be shared between instances, and has a life so long that it’s just easier to allocate it all at once, rather than using up heap or stack for them (in xine-ui, especially, some structures were accessed through a .bss pointer, which was set to an allocated memory area once the feature was activated, and freed either when the feature was temporarily not used anymore, or when the program exited; while you don’t always get the stream information, trying to save a few KiBs of memory by using heap memory might not be a good idea if you have to access the data through a pointer, rather than having the structure in .bss and skipping over the pointer).

So for this reason while I’d be happy if we could find ways to avoid using COW sections at all and still be fast, I’m not targetting moving all the numbers to zero, I just want to make sure that there aren’t memory areas where the space is just wasted.

On the other hand, there are cases which show that something in the design of the program might just be way out of the sane league, for instance the 10MB of .bss that is used by the quota support of xfsprogs is tremendously suspicious.

New patches! Just for our Gentoo users – and for developers of other distributions who want to take the patches ;) – I’ve added a few more patches to my overlay at git://git.flameeyes.eu/overlay.git: giflib’s patch is now in sync with what upstream applied, moving back a constant to variable as needed, while app-arch/libarchive and sys-apps/file got one patch each to reduce their COW memory usage. Both had good results by applying character arrays in structures.

Posted in Gentoo, English | Tags buffers, COW, cowstats, file, giflib, libarchive, libmagic, RubyELF | 2 comments

Single point of mime detection

Posted by Diego "Flameeyes" Pettenò Sat, 26 Jan 2008 00:50:00 GMT

(The title refers to Single point of failure, which vaguely relates to the issue at hand.)

I was talking with Ian (Amarok) and Pino (Okular) tonight, as I fixed a few more mimetypes in xine-lib, about shared-mime-info.

The SMI package, originally written for GNOME, but nowadays also used by XFCE and KDE4, contains an XML file which contain the list of mime types, the extensions they can take and a few magic numbers to identify the mimetype of a given file.

Having a shared package is quite a good idea actually, and as shared-mime-info is extensible by adding an extra xml file (like OpenOffice does), it adds flexibility to be able to change the data in one place to fetch it from everywhere.

But, one tool that is not using this data is the main consumer of similar data: file(1). The file implementation we use at the moment has its own data files to discover the type of files, it does not use shared-mime-info.

So if anybody would like to cut his teeth with a new application, I suppose it would be at least a nice proof of concept to have a 100% compatible file(1) implementation – libmagic included – which uses shared-mime-info database for gathering the data.

I’m just not sure if shared-mime-info contains enough data to implement that, as file(1) reports a few extra information about the files, like the version they implement (for audio files for instance). It would be nice, though, if we could do that, as that would mean that we’d have a viable single source for such data, which is a good thing as you can share it between even widely different software.

Posted in English | Tags file, GNOME, KDE, KDE4, libmagic, SharedMimeInfo, XFCE | 1 comment

Sometimes the come back

Posted by Diego "Flameeyes" Pettenò Fri, 25 Jan 2008 18:27:00 GMT

Sometimes things from your past might come back to your present. They have been waiting in your future for a long time, just to make you wonder how the hell did you met them once again.

No, not talking about anything philosophical here. While I had some of these encounters before (for instance like meeting an ex-classmate after about ten years when doing driving practices).

Today kernel 2.6.24 was released, and during make oldconfig, I’ve seen support for the Winchiphead (WCH) CH341 USB to serial adapter. Which is the chip found on one of the four USB to serial adapters I bought from Hong-Kong last year and for which I wanted to try writing a driver . (see also this).

After almost an year, I gained a new USB to serial adapter. This is not a bad thing to me.

Posted in English | Tags drivers, kernel, Serial, USB, WCH | no comments

A good reason to run autotools during src_unpack

Posted by Diego "Flameeyes" Pettenò Fri, 25 Jan 2008 17:20:00 GMT

I’ve seen a lot of people thinking that the split between src_unpack and src_compile functions is mostly useless, as they can’t find any good reason for not patching during src_compile.

There are, though, a few ineresting and useful reasons for that, which I’ll try to explain here so that they can be put up to debate if needed, and at least unofficially documented in public.

The first obvious reason is that, if I am to check someone else’s package for a bug that I might be involved in (for instance if I maintain the backend library causing the failure), the easyest way for me to look at the source where the error is would be to just run ebuild foo-1.ebuild unpack and then check the source out. If patches are applied during src_compile phase rather than src_unpack, then I’d be looking at the wrong sources, maybe looking at the wrong line number (this actually happened to me a couple of times before).

But this is true even for non-human tests. Take for instance the default-bsd profile. That profile has a very simple check for a very broken source file, that would compile, and then fail at runtime. This is very unfortunate, but might be very widely used because it’s part of gnulib. The test is done at the start of the compile phase, so to at least catch most of the use cases.

If a software has a broken gnulib, and the patch to fix that was applied during compile phase, then the test will trigger false positives, as the patches wouldn’t be applied, yet. Even worse, if the software is not using gnulib out of the box, but the support for gnulib is patched in, for instance because it’s used by an half-fork by Debian – metamail comes to mind -, and the patch is applied in src_unpack, then the sanity check will let the build compile, while it shouldn’t have.

But this relates to patch, while my entry titles “A good reason to run autotools during src_unpack”. In truth, what I said applies to autotools exactly for the same reason.

You can’t really do any sanity check using bashrc on autotools at the moment because a lot of packages re-run them during src_compile phase rather than during the src_unpack phase, so finding known-broken packages is difficult. I’ve been thinking about resuming the work on autoepatch, but for that to work, we’d have to be able to be able to properly check after src_unpack, and expect not to have autotools ran afterward.

At the moment one of the biggest violators of this idea is KDE, and that’s also partly my fault for not looking for a better way to handle it. But beside that, if you know of anything that runs autotools during src_compile, it’s a good idea to submit patches at least to the latest ebuilds to move the run in src_unpack at least. Same for stuff patching during src_compile.

I suppose one way to fix this would be to start launching QA warnings about use of autotools.eclass functions during non-unpack phases, as well as epatch. Or we could try with developer-designed bashrc entries that mark all the source file as read-only after src_unpack phase.

Suggestions about this are welcome as usual. More work will certainly follow.

Update: I’ve talked with Halcy0n, the current QA team leader, and I’ve now added a warning when the autotools.eclass functions are ran outside the unpack phase. I suppose I should say to feel free to report bugs about those ebuilds from now on.

Posted in Gentoo, English | Tags Autoepatch, autotools, epatch, QA | 3 comments

A few little updates

Posted by Diego "Flameeyes" Pettenò Thu, 24 Jan 2008 17:34:00 GMT

First of all I want to thank all the users who gave me feedback with my last blog post :) Now that is something that really improves the morale of developers!

Now, I’m spending ore time on finishing the text of the course I’m going to teach, afterward I’ll resume on looking for new areas that are not usually covered by software developers.

I also decided that I have to find a more stable job, and hopefully one not hurting my time to work on Free Software. In particular, I’ve contacted the publisher for which I I worked for a few years ago to see if they have any translation I might be able to do; working as a translator for books isn’t that bad, especially since it tends to let me concentrate more on concepts which I might just skim through otherwise, and it pays well.

I’m really waiting for the Sony Reader to arrive, it should arrive soon enough, although the Italian postal service sucks a lot. I’m still collecting PDF files to load once I receive it.

I also found a likely way to get rid of the extra stress that could cause me to burn if not relieved: with a friend of mine we’re going to try organizing a campaign for an RPG. We haven’t decided the details yet, but we’re likely to go through MSN for that. I have prepared a script for Adium (on OSX) for dice rolling, I’ll see to publish that soon enough.

On a more Gentooish note, since last night there’s PulseAudio 0.9.9 in tree. This version is just a security fix over 0.9.8, but the -r1 version in ~arch now has a glib USE flag, enabled by default, that allows you to remove glib support, for instance for embedded systems. At the moment glib is used only by the PulseAudio management software (paman, pavumeter, pavucontrol, paprefs padevchooser); so if you’re not using those, you might disable the glib USE flag mostly safely, for now.

I also did a quick change to PulseAudio today to reduce the RSS, but I haven’t had time to dig deeper, as today was a busy busy day.

Posted in Gentoo, English | Tags Adium, Job, PulseAudio, RPG, StressRelieve | 1 comment

Older posts: 1 2 3 4

Still thanking all the people who helped me with donations for the new disks for Enterprise and generic maintenance costs, I decided to change direction with respect to donations.

While I still accept gifts from my Amazon wishlist, I'd rather suggest, to anybody who wants to help me to instead donate to one of the following organisations:

Fondazione Italiana per la ricerca sulle Malattie del Pancreas (FIMP): donations (IBAN IT 70 J 02008 11709 000040424060 for non Italian-speakers) and 5‰ (for Italian people)
National (USA) Pancreas Foundation. (for Americans, a 501(c)(3); if you want to let me know, use the in Honor donation option, and use this address: Diego Pettenò - Via Scaramuzza, 125 - 30174 Mestre, VE - Italy.)

I'm asking this because after my hospitalisation I see little point in saving money for the future, and research into Pancreatic diseases is more likely to make me good.

Thanks in advance to everybody wanting to contribute.

Syndicate

del.icio.us/flameeyes

Jul 03

Sito ufficiale PlayStation 3

Jun 29

Gentoo PAM maintenance and documentation.
RubyTag++
Rust, a Ruby extensions generator
SerielleKonsole: serial terminal emulator for KDE

Flameeyes's Weblog

Proud to be European

Random notes about xine plugins interface

Another power outage

More about Reader and PDFs

My impressions about the Reader

Some notes about multi-threading

A good kind of cow

Single point of mime detection

Sometimes the come back

A good reason to run autotools during src_unpack

A few little updates

Donations and gifts

Archives

Categories

Syndicate

Syndicate

del.icio.us/flameeyes

Working on...