Use reproducible global sums in DGLC when bfbflag is set

[billsacks]

Description of changes

The global sums in DGLC lead to roundoff-level differences with different processor counts. This causes failures in tests - like the ERP test - that check for bit-for-bit results with changing process count.

This PR uses the bfbflag attribute to choose whether to perform global sums in the original way or using shr_reprosum_mod to get bit-for-bit results when changing processor count. This bfbflag attribute is added in ESCOMP/CMEPS#704, and the changes in this PR are connected to that CMEPS PR. By default we still use the original way, both for better performance and because UFS doesn't have shr_reprosum_mod. (Note that an error will be raised if trying to set bfbflag with UFS.) Some CIME tests, such as the ERP test, turn on BFBFLAG so will leverage this new behavior.

The global sums routine is duplicated between CMEPS and CDEPS. I don't like that. But I couldn't see a good alternative. Alternatives I considered were:

• Have cmeps use the version from cdeps. However, this both (a) seems like it might not work in all build environments, and (b) feels like a weird dependency
• Putting this in CESM_share. But then, since UFS doesn't use CESM_share, we would need a wrapper on the UFS side that would involve about the same amount of duplication that we currently have.

Most of the changes were written by Claude, but with detailed guidance and careful reviews by myself.

Specific notes

Contributors other than yourself, if any:

CDEPS Issues Fixed (include github issue #):

Are there dependencies on other component PRs (if so list):

• Depends on Use reproducible global sums when BFBFLAG is set CMEPS#704

Are changes expected to change answers (bfb, different to roundoff, more substantial): roundoff-level changes for tests that set BFBFLAG and use DGLC (ERP, PEM, PET, SEQ)

Any User Interface Changes (namelist or namelist defaults changes):

Testing performed (e.g. aux_cdeps, CESM prealpha, etc):

aux_cime_baselines and aux_cdeps in the context of cesm3_0_beta09, with this branch and the CMEPS branch in ESCOMP/CMEPS#704.

All tests pass and are bit-for-bit except expected failures and this test that has roundoff-level changes due to a change in CMEPS: SMS_Ly2.f09_g17_gris20.T1850Gg.derecho_intel.

Also some additional tests to give better test coverage; these included the following (along with some others that were less good tests, so I'm not documenting them here)

• SMS_Ld3.ne30pg3_t232.B1850C_LTso.derecho_intel.drv-glc_avg_frequently - bit-for-bit with baseline
• ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly - roundoff-level diffs from baseline; bit-for-bit with baseline if I set BFBFLAG FALSE.

[billsacks]

I have done some additional testing, which I have documented in the top-level comment of this PR.

[Katetc]

HI Bill, this one is simpler. I didn't do a diff against the global_sums_mod in CMEPS but I'm sure they are very similar, so that helps. I think this one can be merged without issue.

[billsacks]

HI Bill, this one is simpler. I didn't do a diff against the global_sums_mod in CMEPS but I'm sure they are very similar, so that helps. I think this one can be merged without issue.

Thanks a lot. I diffed them... the differences are trivial.