On UNIXoid operating systems, you can open(2) a file in many modes. On some operating systems, one of them is O_DIRECT, which stands for direct i/o without any caching. To sum it all up:
The whole notion of "direct IO" is totally braindamaged. Just say no. --Linus Torvalds
Accessing files that were opened with O_DIRECT requires aligned buffers for reading and writing. For example, on Linux 2.6, all buffers must be aligned to 512 bytes and reads and writes can only happen in multiples of 512 bytes. It's fairly easy to align your data to 512 bytes, though.
On Linux 2.4, on the other hand, buffers have to be aligned to multiples of the underlying file system's logical block size - which is generally much larger than 512 bytes. Size also has to be a multiple of the block size. It's not so easy to solve this generally. It's also often forgotten because nobody wants to use Linux 2.4 anymore, at least not for development work.
We ran into this problem at least three times in three different places in 2007. You can tell that I work for a database company.