Can not initialize Cdo object for CDO version 1.9.10

[pjpetersik]

I am working with the python CDO binding in a docker container which runs a debian bullseye distribution (python:3.9.15-slim-bullseye). I installed the cdo biniary using apt-get install cdo which installs CDO version 1.9.10 (https://packages.debian.org/bullseye/cdo).

When I call Cdo() I get an IndexError with a Traceback that ends as follows:

  File "/usr/local/lib/python3.9/site-packages/cdo.py", line 190, in __init__
    self.operators         = self.__getOperators()
  File "/usr/local/lib/python3.9/site-packages/cdo.py", line 341, in __getOperators
    operators[op] = int(ios[i][1:len(ios[i]) - 1].split('|')[1])
IndexError: list index out of range

This is caused by an unexpected behaviour of cdo --operators. When running cdo --operators in the command line, I get an unexpected output in the first line of the output:

PRE-MAIN-DEBUG Registering library [eckit] with address [0x7fa77f2ad9a0]
abs              Absolute value                                                            (1|1)
acos             Arc cosine                                                                (1|1)
add              Add two fields                                                            (2|1)
addc             Add a constant                                                            (1|1)
addtrend         Add trend                                                                 (3|1)
...

So the first line of the output is actually not an operator. I can monkey patch this error with the following code which basically jumps the first line of the output of cdo --operators:

import os
import subprocess

from cdo import Cdo

class CdoMonkeyPatch(Cdo):
    def __getOperators(self):  # {{{
        operators = {}
        proc = subprocess.Popen([self.CDO, '--operators'],
                                stderr=subprocess.PIPE, stdout=subprocess.PIPE)
        ret = proc.communicate()
        ops = list(map(lambda x: x.split(' ')[0], ret[0].decode(
            "utf-8")[0:-1].split(os.linesep)))
        ios = list(map(lambda x: x.split(' ')
                   [-1], ret[0].decode("utf-8")[0:-1].split(os.linesep)))
        for i, op in enumerate(ops):
            if i != 0:
                operators[op] = int(ios[i][1:len(ios[i]) - 1].split('|')[1])

        return operators  # }}}

Is there something wrong with my CDO installation?

BTW: This error did not appear when I was using the python:3.8.13-slim-buster image for which CDO version 1.9.6 is installed by apt-get install cdo (https://packages.debian.org/buster/cdo).

[mathisc]

Hi,

I ran into the same error and I dug a little bit more to get a better understanding :
Previously in my image, the default cdo version was 1.9.6, it was recently updated to 1.9.10...
In 1.9.10, with DEBUG=True flag in your environment, the first value for operator is PRE_MAIN_DEBUG, this was not the case for 1.9.6.

Here is a little script explaining the issue :

import os
import subprocess

def print_first_operators():
    proc = subprocess.Popen(["cdo", '--operators'],
                            stderr=subprocess.PIPE, stdout=subprocess.PIPE)
    ret = proc.communicate()
    ops = list(map(lambda x: x.split(' ')[0], ret[0].decode(
        "utf-8")[0:-1].split(os.linesep)))
    ios = list(map(lambda x: x.split(' ')[-1], ret[0].decode(
        "utf-8")[0:-1].split(os.linesep)))

    print(f'{ops[0]} {ios[0]}')

os.environ['DEBUG'] = "False"
print_first_operators()
# prints:
# cdo-1.9.10 : abs (1|1)
# cdo-1.9.6 : abs (1|1)
os.environ['DEBUG'] = "True"
print_first_operators()
# prints :
# cdo-1.9.10 : PRE-MAIN-DEBUG [0x7ff96caac9a0]
# cdo-1.9.6 : abs (1|1)

[mathisc]

(NB: created merge request for that : #53 )

[Try2Code]

hi!
Thx for the report. But it's hard to provide workarounds for strange installations. There is no direct link from CDO to eckit. Someone published a very strange build of CDO. I doubt that this is coming from debian since they use a very strict building policy.

Can you tell mode about which image exactly is affected?

[Try2Code]

hi again!
I investigated a bit more in to this error. Unfortunately all operators which write to stdout (showname, showlevel, sinfov, etc.) have problems.
So I recommend to avoid the debian (or debian-based) binary. for linux conda (conda-force channel) is a valid options. Those releases are update-to-date. you can check at https://code.mpimet.mpg.de/projects/cdo/wiki/Linux_Platform for more info

[pjpetersik]

Hi @Try2Code,
thanks for having a look into the issue. I followed your recommendation and solved the issue on my side by installing CDO from source following this gist: https://gist.github.com/jeffbyrnes/e56d294c216fbd30fd2fd32e576db81c