Pre-Pre-Scriptum: Please, look for the edll library. It is now available as a free library for MinGW.

Pre-Scriptum: I'm currently working on this project, so you will have to wait a little bit before to have a nice web page and files to download. I'm very close to having a working version. C programs already link, but I'd like to have C++ as well so most everyone can use the library. You should see files in this project within 2 or 3 days (April 28, 2005).

What are Enhanced DLLs?

This is actually a project I'm working on to get a much better interface than the Microsoft .DLL scheme. The idea, is to have a libtool ltdl library which can load object files compatible with your processor (i.e. Pentium) and link automatically as necessary. The result is to have linking in both directions and especially to create modules which have undefined symbols which is very important when you don't want to (1) duplicate code a bizillion times and (2) not have a bizillion DLLs around so it actually has a chance to work. Note that will only work with MinGW. Ho... by the way... you won't need to register your modules and you won't need to use the COM interface if you don't want to. Isn't that enhanced?! 8-)

What is this about?

The following is a schema showing what is intended. This works on systems such as Linux, Mac OS/X, SunOS... well, all Unices, you name it. Note that it is not only MS-Windows which doesn't support undefined symbols. BeOS has the same problem and my solution should be applicable to all of these systems which can't do a proper dynamic loading.

The main problem which can't be solved, if you need (want) to have an application which defines one or more symbols necessary for a dynamic link library to properly link, then you need to create a Start executable which loads your Main application. That in turn will automatically load the Library which can then be linked against the Main module.It seems to me that this is a small draw back in comparison to all the possible benefits.

In this schema we see the Start module. This is a standard executable for the target machine. The Start module is limited to whatever the target machine supports. It is not required if your system supports undefined symbols and back linking (i.e. Linux doesn't require it; note however that you will need the -r dynamic command line option to link your main application properly for back linking.)

The Start module will know which modules to load and will do so using the libbfd.a library. Note that this library can load many different object formats such as COFF and ELF. It should be possible to support all the formats that the libbfd.a supports. The libbfd.a knows how to load object files and it can link them together. This can be done at link time using the linker (see ld) or at run time and that's what this project is about. This means instead of using the available system linker, we will create an application dynamic linker which can be used on any system.

Once the Start module loaded the Main module and the necessary Libraries, it can then call some main() function in the Main module. The libbfd.a library can read the Start module first in order to get all the symbols defined in that module. This way, it can back link to itself. In other words, the Main module can make use of the libbfd.a without having to link to it at time of development.

Now that the Main module was loaded, we can have Plug-ins. These are loaded exactly the same way as the Main and Library modules were loaded. The interesting thing here is that all the Plug-ins will have many undefined symbols (contrary to the .DLL scheme). This helps very much in case you have singletons in your library. Also, it reduces the size of the code. Of course, some of the functions in Main could be moved in your .DLL and both, Main and Plug-ins could link to that .DLL. This would resolve the problem of undefined symbols. However, in many applications, this would be a nightmare. (More or less, it would make your whole application much more complicated to program... what most people trying to use COM run into these days!)

Of course, each Plug-in can reference Extra libraries as required. For instance, a browser could load a Plug-in to playback a video. This video playback Plug-in could then load ffmpeg as a library.

Algorithm

At this time, this is what I think will work. I will have to, of course, write it to make it that it indeed does work. It is written in some sort of pseudo language.

	// returns the loaded module unless an error occurs in which case NULL
	LoadModule <filename>

	If ! exists filename then return NULL
	If module already loaded Then
		Add one to the reference counter
		return Module
	End If
	If first time called Then
		Create a list of Modules
	End If
	Create a new module object and link it to the list of modules
	Loop until all modules loaded
		Open module with libbfd.a
		Search sections of interest(1)
		All Sections are added to a list
		If .dynamic_link section isn't empty Then
			Loop for each named .dynamic_link module necessary for this module to load
				If that .dynamic_link module is not already loaded Then
					Create a new module
					Append this new module to our list of modules
				Else
					Add one to the reference counter of exist module
				End If
			End Loop
		End If
	End Loop
	Retrieve the size of each section we want to load
	Allocate a buffer for each section we want to load
	Make the .text sections executable
	Assign the buffer addresses to the different sections in libbfd.a
	Ask libbfd.a to relocate all the modules we just loaded
	Load the relocated section data in our allocated buffers(2)
	If possible on the system, make the memory of all the read-only sections read-only

	Return Module

(1)

The sections of interest are: .text (the actual code to execute) .data (read-write data loaded from file) .bss (read-write data initialized to zero) .rodata (read-only data loaded from file) .dynamic_link (list of modules required to load this module) Notes: The .dynamic_link section may have to be created "by hand" since the C/C++ linker won't create such a section in object files. The .text section is always read-only. If possible, we should change the protection of the memory buffer after we loaded the section data so it is read-only. Under MS-Windows, it is done with the MemoryProtection() function [not too sure about the name, but that's most certainly close to that].

(2)

As an optimization, we don't actually want to load the section data unless it is small. What would be a lot better is to protect the memory buffer against read/write/execute and wait for exceptions. Whenever an exception is thrown (bus error or segmentation fault) then we can load the piece missing. This is what most systems do these days and it makes the loading of dynamic objects really fast. There are, however, some drawbacks since a realtime application may suffer from time to time of an unexpected disk access. Also, the use of such a feature would require multi-thread support in a multi-thread environment. -- I'm not too sure but it is not unlikely that the libbfd.a would load the entire section when we assign the start address of the section. Thought it would be even better to use file mapping, it isn't possible to do so because the section data needs to be relocated and that couldn't be available in this case.

It will be necessary to keep track of the different modules which have been loaded at any time. This means we need a global list of all the modules which have been loaded. This also means we need a reference counter for these modules so when we unload them we know whether we can free a module from memory or not.

NOTE: The first version won't be multi-thread safe (we will need such a version at some point; however, it seems to me that we will just need two versions to make sure we don't force a link with libpthread).

This code requires the following:

• A module structure with:
  • next/previous to link modules together
  • reference counter
  • section type, size, buffer
  • array of buffers to load the sections
  • a list of dynamic-modules (to be able to deference them)
• Test whether a file exists
• Find sections in an object file
• Retrieve section information (name, size)
• Read the dynamic link section for names of other objects to read
• Allocate a buffer which can load a .text section; under MS-Windows, this means using the VirtualAlloc() functions if running Windows 2000 or better; under old versions of MS-Windows, a regular GlobalAlloc() is fine since all the memory is mapped with the execute flag turned on. This allocation needs to be programmed as a separate function to make sure we can take all these cases in account properly. For instance, under Linux you change the memory protection after you allocated it. Under MS-Windows, you specify the memory protection at the time you call VirtualAlloc()
• Do the relocation of the .text and .data sections of all the modules
• Load the sections in the buffers we allocated
• On system where it is possible, change the memory protection scheme to read-only for all the sections which are read only.

	UnloadModule module

	If module is NULL, Return
	Subtract one to the module reference
	If module reference is not zero, Return
	Go through the list of .dynamic_link and unload each of these modules
	Free all the buffers used by this module
	Unlink and free this module

	Return

The unloading of a module is the opposite of the loading. Note that it will decrease the reference counter by one on each call until it reaches zero. When zero is reached, it is removed from memory entirely. This means, of course, that all the functions and data of that module won't be accessible anymore.

References

• libbfd documentation (HTML)