cron job to run under conda virtual environment

Question

I have a Anaconda Python Virtual Environment set up and if I run my project while that virutal environment is activated everything runs great.

But I have a cronjob configured to run it every hour. I piped the output to a log because it wasn't running correctly.

crontab -e:

10 * * * * bash /work/sql_server_etl/src/python/run_parallel_workflow.sh >> /home/etlservice/cronlog.log 2>&1

I get this error in the cronlog.log:

Traceback (most recent call last):
  File "__parallel_workflow.py", line 10, in <module>
    import yaml
ImportError: No module named yaml

That is indicative of the cronjob somehow not running the file without the virtual environment activated.

To remedy this I added a line to the /home/user/.bash_profile file:

conda activate ~/anaconda3/envs/sql_server_etl/

Now when I login the environment is activated automatically.

However, the problem persists.

I tried one more thing. I changed the cronjob, (and I also tried this in the bash file the cronjob runs) to explicitly manually activate the environment each time it runs, but to no avail:

10 * * * * conda activate ~/anaconda3/envs/sql_server_etl/ && bash /work/sql_server_etl/src/python/run_parallel_workflow.sh >> /home/etlservice/cronlog.log 2>&1

Of course, nothing I've tried has fixed it. I really know nothing about linux so maybe there's something obvious I need to change.

So, is there anyway to specify that the cronjob should run under a virutal environment?

Answer 1

Posted a working solution (on Ubuntu 18.04) with detailed reasoning on SO.

The short form is:

1. Copy snippet appended by Anaconda in ~/.bashrc (at the end of the file) to a separate file ~/.bashrc_conda

As of Anaconda 2020.02 installation, the snippet reads as follows:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/USERNAME/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/USERNAME/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/home/USERNAME/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/USERNAME/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Make sure that:

• The path /home/USERNAME/anaconda3/ is correct.
• The user running the cronjob has read permissions for ~/.bashrc_conda (and no other user can write to this file).

2. In crontab -e add lines to run cronjobs on bash and to source ~/.bashrc_conda

Run crontab -e and insert the following before the cronjob:

SHELL=/bin/bash
BASH_ENV=~/.bashrc_conda

3. In crontab -e include at beginning of the cronjob conda activate my_env; as in example

Example of entry for a script that would execute at noon 12:30 each day on the Python interpreter within the conda environment:

30 12 * * * conda activate my_env; python /path/to/script.py; conda deactivate

And that's it.

You may want to check from time to time that the snippet in ~/.bashrc_conda is up to date in case conda updates its snippet in ~/.bashrc.

Answer 2

This is a solution that has worked well for me.

source /root/miniconda3/etc/profile.d/conda.sh && \
conda activate <your_env> && \
python <your_application> &

I am using miniconda with Conda version 4.7.12 on a Ubuntu 18.04.3 LTS.

I am able to place the above inside a script and run it via crontab as well without any trouble.

Answer 3

Found answer on stack over flow:

https://stackoverflow.com/questions/3287038/cron-and-virtualenv

The solution is to reference the python executable within the virtual environment itself. In my case, I changed the bash file to run this executable:

/home/etlservice/anaconda3/envs/sql_server_etl/bin/python

Answer 4

This solution works to me.

Try to use

source /path/to/conda/bin/activate my_env

instead of

conda activate my_env

in your bash script run_parallel_workflow.sh.