Dynamic imports in Python

01 May 2022 - tsp
Last update 01 May 2022
Reading time 4 mins

The problem
The example
- Sample modules
- The main loader
How to load plugins from a directory

The problem

One often wants to dynamically load modules into ones application. In C applications one can imagine loading user supplied dynamic link libraries / shared objects (DLL or SO) into the programs process and execute them - for example to realize plugins or extensions to ones application. This is also often done at runtime - for example using dlopen (Unices) or LoadModule (Windows). Since I had to do this again while extending my lambda running infrastructure to support Python (in a limited way) - but this time not in ANSI C but in Python - and I had to re-read the documentation over and over again I decided to write a short summary. It’s a little bit more complicated to do in Python than with native code but still pretty simple.

Note that with this method there is no separation between the running container and the loaded modules - they’re loaded into the same process and have the same privileges. When loading dynamic resources one should make sure these are coming from a trusted source. In any other case one should consider the approach of launching a separate container process, dropping privileges after opening all allowed resources and then loading the required code - including a Python interpreter - and execute the untrusted payload.

The example

Sample modules

The example modules will just return a value specific to their module type or version. They will all expose the same TestModuleFactory factory class that is capable of instantiating a TestModule that will accept a single parameter to show they’re indeed the expected instances as well as exposes the same getValue method for every of the modules (as expected for a plugin system for example).

For the short example the first test module will be implemented in modules/test1.py:

class TestModuleFactory:
        def getInstance(self, id):
                return TestModule(id)

class TestModule:
        def __init__(self, id):
                self.id = id
        def getValue(self):
                return "First test module (1): {}".format(self.id)

A second test module looking nearly the same will be defined in modules/test2.py:

class TestModuleFactory:
        def getInstance(self, id):
                return TestModule(id)

class TestModule:
        def __init__(self, id):
                self.id = id
        def getValue(self):
                return "Second test module (2): {}".format(self.id)

The main loader

The loading of modules will be realized using importlib.util in three steps:

First the specification for the module will be created (this is specified in PEP 451)
Then the module will be loaded using the importlib
As a last step for module initialization the module will be executed
Then the loading function will query our TestModule class and return an instance

import importlib.util

def loadModuleFromFile(filename):
        spec = importlib.util.spec_from_file_location("testmodule", filename)
        module = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(module)
        return module.TestModuleFactory()

m1 = loadModuleFromFile('./modules/test1.py')
m2 = loadModuleFromFile('./modules/test2.py')

c1 = m1.getInstance(1)
c2 = m2.getInstance(2)
c3 = m1.getInstance(3)
c4 = m2.getInstance(4)

print("Module 1, ID 1: {}".format(c1.getValue()))
print("Module 2, ID 2: {}".format(c2.getValue()))
print("Module 1, ID 3: {}".format(c3.getValue()))
print("Module 2, ID 4: {}".format(c4.getValue()))

Executing this will just return

Module 1, ID 1: First test module (1): 1
Module 2, ID 2: Second test module (2): 2
Module 1, ID 3: First test module (1): 3
Module 2, ID 4: Second test module (2): 4

How to load plugins from a directory

So now let’s assume one just wants to load all Python modules from a given plugin directory. One can easily achieve this using any of the directory iteration methods - for example os.scandir. This example assumes the same module structure in modules as before:

import os
import importlib.util

def loadModules(moduleDirectory):
        mods = []
        with os.scandir(moduleDirectory) as it:
                for entry in it:
                        if entry.name.endswith(".py") and entry.is_file():
                                spec = importlib.util.spec_from_file_location("module", entry.path)
                                module = importlib.util.module_from_spec(spec)
                                spec.loader.exec_module(module)
                                mods.append(module.TestModuleFactory())
        return mods

Now one can use the returned module factory list to create new instances every time one likes to. The following sample would simply generate a bunch of instances and then call their getValue methods to show this really works as expected:

modules = loadModules("./modules")
n = 0

instances = []

for i in range(3):
        for mod in modules:
                n = n + 1
                classInstance = mod.getInstance(n)
                instances.append(classInstance)

for instance in instances:
        print(instance.getValue())