Dynamically create conda envs for your Python scripts

Have you ever wondered how to run some of your scripts with different requirements, but without explicitly having to create envs for them?

Like for example, you would only want to provide pip requirements and that's it?

Below I can show you how you could do that.

Essentially, we will only have to use a bit of subprocess calls and rely on a neat feature that miniconda provides.

The prerequisite is to have miniconda installed in whatever environment you are using.

The below code allows you to easily run the same script, but with different version of the same package.

You should put this code in a file named `my_package/starter.py`.

                                
import json
import subprocess


def create_or_get_env(env_name: str) -> str:
    conda_envs = json.loads(
        subprocess.check_output(
            "conda info -e --json", 
            shell=True, 
            executable="/bin/bash"
        )
    )
    root_prefix = conda_envs["root_prefix"]
    if env_name == "base":
        return root_prefix
    # Naively assume it will always be <base>/envs/<name_of_env>
    # Convert to os.sep
    envs = {
        "/".join(
            x[len(root_prefix)].split("/")[1:]): 
            x for x in conda_envs["envs"] 
        if x != root_prefix
    }
    if env_name not in envs:
        # Assume it is python 3.9.13 only
        creation_json = json.loads(
            subprocess.check_output(
                f"conda create -y -n{env_name} python==3.9.13 --json",
                shell=True,
                executable="/bin/bash"
            )
        )
        # Convert this to a dedicated exception
        assert creation_json["success"]
        return creation_json["prefix"]
    else:
        return envs[env_name]


def install_reqs(env_path: str, reqs: str) -> None:
    # Use some random tempdir
    with open("/tmp/requirements.txt", "w") as f:
        f.write(reqs)
    subprocess.check_output(
        f"{env_path}/bin/python -m pip install -q -r /tmp/requirements.txt", 
        shell=True, 
        executable="/bin/bash"
    )


def run_in_conda_env(env_name: str, reqs: str, python_module: str) -> bytes:
    env_path = create_or_get_env(env_name)
    install_reqs(env_path, reqs)
    return subprocess.check_output(
        f"{env_path}/bin/python -m {python_module}", 
        shell=True, 
        executable="/bin/bash"
    )


def example() -> None:
    requirements = """
    pandas==1.4.3
    """
    print(
        run_in_conda_env(
            "my_test_env", 
            requirements, 
            "my_package.my_script"
        ).decode("ascii")
    )


def example2() -> None:
    requirements = """
    pandas==1.3.5
    """
    print(
        run_in_conda_env(
            "my_test_env2", 
            requirements, 
            "my_package.my_script"
        ).decode("ascii")
    )



if __name__ == "__main__":
    example()
    example2()

You will also need to create the `my_package/my_script.py` file.

The below code should end up in the mentioned file.

                                
import pandas as pd


if __name__ == "__main__":
    print(pd.__version__)
    print(pd.DataFrame([[1, 2, 3]], columns=["A", "B", "C"]))

You can run the above code using `python -m my_module.starter`.

It should create two conda environments with different pandas packages and print the two versions.

The idea is very basic, but one could use it to setup some kind of automatic integration testing or simply ensure some of your scripts always run with the provided requirements.