A site for solving at least some of your technical problems...
A site for solving at least some of your technical problems...
Pre-Pre-Scriptum: Please, look for the edll library. It is now available as a free library for MinGW.
Pre-Scriptum: I'm currently working on this project, so you will have to wait a little bit before to have a nice web page and files to download. I'm very close to having a working version. C programs already link, but I'd like to have C++ as well so most everyone can use the library. You should see files in this project within 2 or 3 days (April 28, 2005).
This is actually a project I'm working on to get a much better interface than the Microsoft .DLL scheme. The idea, is to have a libtool ltdl library which can load object files compatible with your processor (i.e. Pentium) and link automatically as necessary. The result is to have linking in both directions and especially to create modules which have undefined symbols which is very important when you don't want to (1) duplicate code a bizillion times and (2) not have a bizillion DLLs around so it actually has a chance to work. Note that will only work with MinGW. Ho... by the way... you won't need to register your modules and you won't need to use the COM interface if you don't want to. Isn't that enhanced?! 8-)
The following is a schema showing what is intended. This works on systems such as Linux, Mac OS/X, SunOS... well, all Unices, you name it. Note that it is not only MS-Windows which doesn't support undefined symbols. BeOS has the same problem and my solution should be applicable to all of these systems which can't do a proper dynamic loading.
The main problem which can't be solved, if you need (want) to have an application which defines one or more symbols necessary for a dynamic link library to properly link, then you need to create a Start executable which loads your Main application. That in turn will automatically load the Library which can then be linked against the Main module.It seems to me that this is a small draw back in comparison to all the possible benefits.
In this schema we see the Start module. This is a standard executable for the target machine. The Start module is limited to whatever the target machine supports. It is not required if your system supports undefined symbols and back linking (i.e. Linux doesn't require it; note however that you will need the -r dynamic command line option to link your main application properly for back linking.)
The Start module will know which modules to load and will do so using the libbfd.a library. Note that this library can load many different object formats such as COFF and ELF. It should be possible to support all the formats that the libbfd.a supports. The libbfd.a knows how to load object files and it can link them together. This can be done at link time using the linker (see ld) or at run time and that's what this project is about. This means instead of using the available system linker, we will create an application dynamic linker which can be used on any system.
Once the Start module loaded the Main module and the necessary Libraries, it can then call some main() function in the Main module. The libbfd.a library can read the Start module first in order to get all the symbols defined in that module. This way, it can back link to itself. In other words, the Main module can make use of the libbfd.a without having to link to it at time of development.
Now that the Main module was loaded, we can have Plug-ins. These are loaded exactly the same way as the Main and Library modules were loaded. The interesting thing here is that all the Plug-ins will have many undefined symbols (contrary to the .DLL scheme). This helps very much in case you have singletons in your library. Also, it reduces the size of the code. Of course, some of the functions in Main could be moved in your .DLL and both, Main and Plug-ins could link to that .DLL. This would resolve the problem of undefined symbols. However, in many applications, this would be a nightmare. (More or less, it would make your whole application much more complicated to program... what most people trying to use COM run into these days!)
Of course, each Plug-in can reference Extra libraries as required. For instance, a browser could load a Plug-in to playback a video. This video playback Plug-in could then load ffmpeg as a library.
At this time, this is what I think will work. I will have to, of course, write it to make it that it indeed does work. It is written in some sort of pseudo language.
// returns the loaded module unless an error occurs in which case NULL LoadModule <filename> If ! exists filename then return NULL If module already loaded Then Add one to the reference counter return Module End If If first time called Then Create a list of Modules End If Create a new module object and link it to the list of modules Loop until all modules loaded Open module with libbfd.a Search sections of interest(1) All Sections are added to a list If .dynamic_link section isn't empty Then Loop for each named .dynamic_link module necessary for this module to load If that .dynamic_link module is not already loaded Then Create a new module Append this new module to our list of modules Else Add one to the reference counter of exist module End If End Loop End If End Loop Retrieve the size of each section we want to load Allocate a buffer for each section we want to load Make the .text sections executable Assign the buffer addresses to the different sections in libbfd.a Ask libbfd.a to relocate all the modules we just loaded Load the relocated section data in our allocated buffers(2) If possible on the system, make the memory of all the read-only sections read-only Return Module
The sections of interest are:
- .text (the actual code to execute)
- .data (read-write data loaded from file)
- .bss (read-write data initialized to zero)
- .rodata (read-only data loaded from file)
- .dynamic_link (list of modules required to load this module)
- The .dynamic_link section may have to be created "by hand" since the C/C++ linker won't create such a section in object files.
- The .text section is always read-only. If possible, we should change the protection of the memory buffer after we loaded the section data so it is read-only. Under MS-Windows, it is done with the MemoryProtection() function [not too sure about the name, but that's most certainly close to that].
As an optimization, we don't actually want to load the section data unless it is small. What would be a lot better is to protect the memory buffer against read/write/execute and wait for exceptions. Whenever an exception is thrown (bus error or segmentation fault) then we can load the piece missing. This is what most systems do these days and it makes the loading of dynamic objects really fast. There are, however, some drawbacks since a realtime application may suffer from time to time of an unexpected disk access. Also, the use of such a feature would require multi-thread support in a multi-thread environment. -- I'm not too sure but it is not unlikely that the libbfd.a would load the entire section when we assign the start address of the section.
Thought it would be even better to use file mapping, it isn't possible to do so because the section data needs to be relocated and that couldn't be available in this case.
It will be necessary to keep track of the different modules which have been loaded at any time. This means we need a global list of all the modules which have been loaded. This also means we need a reference counter for these modules so when we unload them we know whether we can free a module from memory or not.
NOTE: The first version won't be multi-thread safe (we will need such a version at some point; however, it seems to me that we will just need two versions to make sure we don't force a link with libpthread).
This code requires the following:
UnloadModule module If module is NULL, Return Subtract one to the module reference If module reference is not zero, Return Go through the list of .dynamic_link and unload each of these modules Free all the buffers used by this module Unlink and free this module Return
The unloading of a module is the opposite of the loading. Note that it will decrease the reference counter by one on each call until it reaches zero. When zero is reached, it is removed from memory entirely. This means, of course, that all the functions and data of that module won't be accessible anymore.