Negative array sum in HEMCO #2736
Comments
|
I can't get a HEMCO diagnostic file to upload, but there are no negative values in the HEMCO diagnostic for anthropogenic NO over Thailand.
|
|
Thanks @ktravis213, The number Also, I'm curious about the gray areas in the plot. Are those the negatives? |
|
@ktravis213: I looked at the files you sent. The mask file is OK. The emissions file has all zeros for Solvents and Shipping sectors. If you try excluding those do you still get the error? |
|
@yantosca the minimum value in the data is 0.0E+00. Good idea on solvents and shipping, but I turned them off and still get the error. I also tried setting all the data including zeros in the dataset to float64 and still get this error. Anything else I could try? I noticed this array sum issue doesn't happen with other inventories. |
yantosca
added
topic: Emissions
|
@ktravis213 have you tried adding the emissions without the mask file? Just to see if that's the issue. I wouldn't think so but you never know. |
|
I took the mask off and still get the negative sum. Since CEDS reads in without this issue, I must be doing something wrong in the file itself. HEMCO WARNING: File units do not match: kg m-2 s-1 vs. kg/m2/s. File: /discover/nobackup/ktravis1/data/ExtData/HEMCO/CEDS/v2021-06/2019/NO-em-anthro_CMIP_CEDS_2019.nc
--> LOCATION: HCOIO_READ (HCOIO_READ_STD_MOD.F90)
Unit conversion settings:
- Year, month : 2019 2
Data was in units of kg m-2 s-1 - unit conversion factor is 1.00000000000000
==> Use map_a2a regridding
HEMCO: Leaving HCOIO_READ (HCOIO_READ_STD_MOD.F90) ( 5)
HEMCO: Leaving HCOIO_DATAREAD (HCOIO_DATAREAD_MOD.F90) ( 4)
HEMCO: Entering tIDx_Assign (HCO_TIDX_MOD.F90) ( 4)
HEMCO: Leaving tIDx_Assign (HCO_TIDX_MOD.F90) ( 4)
HEMCO: Entering EmisList_Pass (HCO_EMISLIST_MOD.F90) ( 4)
HEMCO: Entering EmisList_Add (HCO_EMISLIST.F90) ( 5)
HEMCO: Entering Add2EmisList (HCO_EMISLIST_MOD.F90) ( 6)
HEMCO: Leaving Add2EmisList (HCO_EMISLIST_MOD.F90) ( 6)
Container added to EmisList:
Container CEDS_NO_TRA
-->Data type : 1
-->Container ID : 11
-->Target ID : 11
-->File data home? 1
-->Source file : $ROOT/CEDS/v2021-06/$YYYY/NO-em-anthro_CMIP_CEDS_$YYYY.nc
-->ncRead? T
-->Shared data file? F
-->Source parameter: NO_tra
-->Year range : 1750 2019
-->Month range : 1 12
-->Day range : 1 1
-->Hour range : 0 0
-->SpaceDim : 2
-->Array dimension : 144 91
-->Array sum : 1.5103870E-08
-->Array min & max : 0.0000000E+00 1.4178292E-10
-->Time dimension : 1
-->Delta t[h] : 0
-->Local time? F
-->Tempres : Constant
-->OrigUnit : kg/m2/s
-->Concentration? F
-->Coverage : -999
-->Extension Nr : 0
-->Species name : NO
-->HEMCO species ID: 207
-->Category : 1
-->Hierarchy : 5
-->2D emitted into : 1.00000000000000 and 1.00000000000000
HEMCO: Leaving EmisList_Add (HCO_EMISLIST.F90) ( 5)
HEMCO: Leaving EmisList_Pass (HCO_EMISLIST_MOD.F90) ( 4)
HEMCO: Entering HCOIO_DATAREAD (HCOIO_DATAREAD_MOD.F90) ( 4)
HEMCO: Entering HCOIO_READ (HCOIO_READ_STD_MOD.F90) ( 5) |
|
Thanks @ktravis213. I'll take a look at the file again. |
|
Hi @ktravis213. I ran the For the emissions file: The following items DO NOT ADHERE to the COARDS standard:
---------------------------------------------------------------------------
Variable "time" is of type "int" (problem for GCHP)
-> Power:long_name (or Power:standard_name) is missing
-> Residential:long_name (or Residential:standard_name) is missing
-> Industry:long_name (or Industry:standard_name) is missing
-> Solvents:long_name (or Solvents:standard_name) is missing
-> Transportation:long_name (or Transportation:standard_name) is missing
-> Agriculture:long_name (or Agriculture:standard_name) is missing
-> Shipping:long_name (or Shipping:standard_name) is missing
-> The "conventions" global attribute is missing
-> The "history" global attribute is missing
-> The "title" global attribute is missing
The following optional items are RECOMMENDED:
---------------------------------------------------------------------------
-> Consider adding Power:_FillValue
-> Consider adding Power:missing_value
-> Consider adding Residential:_FillValue
-> Consider adding Residential:missing_value
-> Consider adding Industry:_FillValue
-> Consider adding Industry:missing_value
-> Consider adding Solvents:_FillValue
-> Consider adding Solvents:missing_value
-> Consider adding Transportation:_FillValue
-> Consider adding Transportation:missing_value
-> Consider adding Agriculture:_FillValue
-> Consider adding Agriculture:missing_value
-> Consider adding Shipping:_FillValue
-> Consider adding Shipping:missing_value
-> Consider adding the "format" global attribute
-> Consider adding the "references" global attributeFor the mask file: The following items DO NOT ADHERE to the COARDS standard:
---------------------------------------------------------------------------
-> mask:units = "unitless"
-> The "conventions" global attribute is missing
-> The "history" global attribute is missing
-> The "title" global attribute is missingTry running this script on the files and see if you can get it to run without the negative sums: #!/bin/bash
ncatted -a long_name,Power,o,c,"Power" NOx_201902.nc
ncatted -a long_name,Residential,o,c,"Residential" NOx_201902.nc
ncatted -a long_name,Industry,o,c,"Industry" NOx_201902.nc
ncatted -a long_name,Solvents,o,c,"Solvents" NOx_201902.nc
ncatted -a long_name,Transportation,o,c,"Transportation" NOx_201902.nc
ncatted -a long_name,Agriculture,o,c,"Agriculture" NOx_201902.nc
ncatted -a long_name,Shipping,o,c,"Shipping" NOx_201902.nc
ncatted -a conventions,global,o,c,"COARDS" NOx_201902.nc
ncatted -a units,mask,o,c,"1" thailand_mask.01x01.nc
ncatted -a conventions,global,o,c,"COARDS" NOx_201902.nc
|
|
I still get a negative array sum of the same amount in the HEMCO log file after these fixes even though the file looks fine. Any other ideas? |
|
Thanks @ktravis213. Have you tried reading the file into Python (with xarray)? Then you can look for negatives or NaNs. I wonder if there is just still something off about the files even though they look OK at first glance. |
|
Hi @ktravis213, I also noticed that the lon/lat values are irregular: lat = 5.65, 5.75, 5.85, 5.95, 6.05, 6.15, 6.25, 6.35, 6.45, 6.55, 6.65,
6.75, 6.85, 6.95, 7.05, 7.15, 7.24999999999999, 7.34999999999999,
7.44999999999999, 7.54999999999999, 7.64999999999999, 7.74999999999999,
7.84999999999999, 7.94999999999999, 8.04999999999999, 8.14999999999999,
8.24999999999999, 8.34999999999999, 8.44999999999999, 8.54999999999999,
8.64999999999999, 8.74999999999999, 8.84999999999999, 8.94999999999999,
9.04999999999999, 9.14999999999999, 9.24999999999999, 9.34999999999999,
9.44999999999999, 9.54999999999999, 9.64999999999999, 9.74999999999999,
9.84999999999999, 9.94999999999999, 10.05, 10.15, 10.25, 10.35, 10.45,
10.55, 10.65, 10.75, 10.85, 10.95, 11.05, 11.15, 11.25, 11.35, 11.45,
11.55, 11.65, 11.75, 11.85, 11.95, 12.05, 12.15, 12.25, 12.35, 12.45,
12.55, 12.65, 12.75, 12.85, 12.95, 13.05, 13.15, 13.25, 13.35, 13.45,
13.55, 13.65, 13.75, 13.85, 13.95, 14.05, 14.15, 14.25, 14.35, 14.45,
14.55, 14.65, 14.75, 14.85, 14.95, 15.05, 15.15, 15.25, 15.35, 15.45,
15.55, 15.65, 15.75, 15.85, 15.95, 16.05, 16.15, 16.25, 16.35, 16.45,
16.55, 16.65, 16.75, 16.85, 16.95, 17.05, 17.15, 17.25, 17.35, 17.45,
17.55, 17.65, 17.75, 17.85, 17.95, 18.05, 18.15, 18.25, 18.35, 18.45,
18.55, 18.65, 18.75, 18.85, 18.95, 19.05, 19.15, 19.25, 19.35, 19.45,
19.55, 19.6499999999999, 19.75, 19.85, 19.9499999999999,
20.0499999999999, 20.1499999999999, 20.25, 20.3499999999999,
20.4499999999999 ;
lon = 97.25, 97.35, 97.45, 97.55, 97.65, 97.75, 97.85, 97.95, 98.05,
98.1499999999999, 98.2499999999999, 98.3499999999999, 98.4499999999999,
98.5499999999999, 98.6499999999999, 98.7499999999999, 98.8499999999999,
98.9499999999999, 99.0499999999999, 99.1499999999999, 99.2499999999999,
99.3499999999999, 99.4499999999999, 99.5499999999999, 99.6499999999999,
99.7499999999999, 99.8499999999999, 99.9499999999998, 100.05, 100.15,
100.25, 100.35, 100.45, 100.55, 100.65, 100.75, 100.85, 100.95, 101.05,
101.15, 101.25, 101.35, 101.45, 101.55, 101.65, 101.75, 101.85, 101.95,
102.05, 102.15, 102.25, 102.35, 102.45, 102.55, 102.65, 102.75, 102.85,
102.95, 103.05, 103.15, 103.25, 103.35, 103.45, 103.55, 103.65, 103.75,
103.85, 103.95, 104.05, 104.15, 104.25, 104.35, 104.45, 104.55, 104.65,
104.75, 104.85, 104.95, 105.05, 105.15, 105.25, 105.35, 105.45, 105.55 ;These are due to roundoff errors but I wonder if this is causing the issue. |
|
Another good idea @yantosca but unfortunately I still get the same negative sum. I finally just wrote out the entire variable in the log file and this is what it looks like. Some but not all?? of the zeros are being read in as -9.9999998E+30. Any idea why this would be? I tried setting the _FillValue to 0 or removing it and no luck. -9.9999998E+30 -9.9999998E+30 -9.9999998E+30 -9.9999998E+30 -9.9999998E+30 |
|
I emailed out of scope here to try converting the double precision emissions to single-precision. If that doesn't help, could you try reading in an offline HEMCO where the output grid is the same as the input grid? I'm trying to understand if this is happening during the read or during the interpolation to the target grid. |
|
Thanks @ktravis213. I wrote a quick script: #!/usr/bin/env python3
import numpy as np
import xarray as xr
dset = xr.open_dataset("NOx_201902v2.nc")
varnames = dset.data_vars.keys()
for var in varnames:
data = dset[var].values
print(f"{var}: min={np.min(data)} max={np.max(data)}")but this didn't show any negatives: (gcpy_env) $ python -m nox
Power: min=0.0 max=2.012451691056023e-09
Residential: min=0.0 max=1.625734232784585e-11
Industry: min=0.0 max=1.5168748043359952e-09
Solvents: min=0.0 max=0.0
Transportation: min=0.0 max=1.258050672518564e-08
Agriculture: min=0.0 max=2.9642066692580566e-11
Shipping: min=0.0 max=0.0I also wrote a code to read the file into Fortran, but I didn't see any ###### Power ######
### minval: 0.0000000000000000
### maxval: 2.0124516910560229E-009
### sum : 3.5793378796846202E-008
###### Residential ######
### minval: 0.0000000000000000
### maxval: 1.6257342327845851E-011
### sum : 5.9063422323227971E-009
###### Industry ######
### minval: 0.0000000000000000
### maxval: 1.5168748043359952E-009
### sum : 2.6995736361477260E-008
###### Solvents ######
### minval: 0.0000000000000000
### maxval: 0.0000000000000000
### sum : 0.0000000000000000
###### Transportation ######
### minval: 0.0000000000000000
### maxval: 1.2580506725185640E-008
### sum : 1.8297934447896774E-007
###### Agriculture ######
### minval: 0.0000000000000000
### maxval: 2.9642066692580566E-011
### sum : 4.2397109756319829E-009
###### Shipping ######
### minval: 0.0000000000000000
### maxval: 0.0000000000000000
### sum : 0.0000000000000000 So I wonder if the negatives are coming in during the regridding. |
|
@barronh @ktravis213: If you can convert the double precision variables to single precision that might help. You probably don't need double precision for the emissons as they are well within the limits of single precision. |
|
@ktravis213: You can convert the data variables to float with: $ cdo -b f64 copy NOx_201902v2.nc NOx_201902v3.ncThe index variables (lon, lat, time) will still be double. |
|
Hi @yantosca. You are right it is the regridding step. I wrote out the array before and after the regridding step. Unfortunately changing the precision hasn't made the regridding work better. |
Your name
Katie Travis
Your affiliation
NASA LaRC
What happened? What did you expect to happen?
I am implementing a new Thai emissions inventory into GEOS-Chem, and am having trouble getting GEOS-Chem to read the files in correctly. The HEMCO verbose output gives me this - where the array sum is negative. However there are no negative numbers in the file, and no negative fill or missing values set.
What are the steps to reproduce the bug?
Here is my HEMCO_Config.rc file. The NOx emissions file and Thai mask file are attached with .txt extensions.
HEMCO_Config.txt
Please attach any relevant configuration and log files.
NOx_201902.txt
thailand_mask.01x01.txt
What GEOS-Chem version were you using?
14.5.0
What environment were you running GEOS-Chem on?
Local cluster
What compiler and version were you using?
ifort 2021.4.0
Will you be addressing this bug yourself?
Yes
In what configuration were you running GEOS-Chem?
GCClassic
What simulation were you running?
Full chemistry
As what resolution were you running GEOS-Chem?
2x25
What meterology fields did you use?
MERRA-2
Additional information
No response
The text was updated successfully, but these errors were encountered: