Preprocess¶

In this notebook we will preprocess the observations and simulations (UNSEEN).

##Load pacakages
import xarray as xr
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

##This is so variables get printed within jupyter
from IPython.core.interactiveshell import InteractiveShell 
InteractiveShell.ast_node_interactivity = "all"

Observations¶

Let’s download the observations. We do this using the CABra dataset and tools. See Almagro et al. (2021). Using the shiny app you can explore the available observation stations. We select them and download them into this repository using the wget command

# !wget -O ../Data/Obidos_record_CABra.txt https://thecabradataset.shinyapps.io/CABra/_w_dc7ddebc/session/f594ed5c6efaa6a6431e2fcdc0b525ee/download/downloadData?w=dc7ddebc

# !wget -O ../Data/Xingu_record_CABra.txt https://thecabradataset.shinyapps.io/CABra/_w_d75db33b/session/9854b2d5ed51bb88a3fe58c9cae6392c/download/downloadData?w=d75db33b

# !wget -O ../Data/Tapajos_record_CABra.txt https://thecabradataset.shinyapps.io/CABra/_w_d75db33b/session/9854b2d5ed51bb88a3fe58c9cae6392c/download/downloadData?w=d75db33b

Here we load the downloaded datasets. We skip the first rows that provide additional information

Obidos_record = pd.read_csv('../Data/Obidos_record_CABra.txt', sep='\t',skiprows = 8)#, header=[0,1]) # Downloaded from BACra. Skip the header
# Obidos_record

Xingu_record = pd.read_csv('../Data/Xingu_record_CABra.txt', sep='\t',skiprows = 8)#, header=[0,1]) # Downloaded from BACra. Skip the header
# Xingu_record

Tapajos_record = pd.read_csv('../Data/Tapajos_record_CABra.txt', sep='\t',skiprows = 8)#, header=[0,1]) # Downloaded from BACra. Skip the header
# Tapajos_record

Then we need to do some pre-processing. First we name the columns so they can be used to create a datetime format. Then we remove the first row because that provides the units (m3/s) and not values. Third we create the datetime format and set it as index. Finally, we set the streamflow value type to float.

Obidos_record.columns = ['Year', 'Month', 'Day', 'Streamflow', 'Quality']
Obidos_record = Obidos_record[1:]
date = pd.to_datetime(Obidos_record[['Year', 'Month', 'Day']])
Obidos_record.index = date
Obidos_record['Streamflow'] = Obidos_record['Streamflow'].astype(float)
Obidos_record

Xingu_record.columns = ['Year', 'Month', 'Day', 'Streamflow', 'Quality']
Xingu_record = Xingu_record[1:]
Xingu_record.index = date #same date as Obidos
Xingu_record['Streamflow'] = Xingu_record['Streamflow'].astype(float)
Xingu_record

Tapajos_record.columns = ['Year', 'Month', 'Day', 'Streamflow', 'Quality']
Tapajos_record = Tapajos_record[1:]
Tapajos_record.index = date #same date as Obidos
Tapajos_record['Streamflow'] = Tapajos_record['Streamflow'].astype(float)
Tapajos_record

	Year	Month	Day	Streamflow	Quality
1980-10-01	1980	10	1	94579.891	2
1980-10-02	1980	10	2	94057.414	2
1980-10-03	1980	10	3	93189.023	2
1980-10-04	1980	10	4	92669.445	2
1980-10-05	1980	10	5	91805.898	2
...	...	...	...	...	...
2010-09-26	2010	9	26	97208.531	2
2010-09-27	2010	9	27	95978.469	2
2010-09-28	2010	9	28	94405.609	2
2010-09-29	2010	9	29	92323.664	2
2010-09-30	2010	9	30	90259.180	2

10957 rows × 5 columns

	Year	Month	Day	Streamflow	Quality
1980-10-01	1980	10	1	995.854	2
1980-10-02	1980	10	2	1011.401	2
1980-10-03	1980	10	3	1011.401	2
1980-10-04	1980	10	4	1027.081	2
1980-10-05	1980	10	5	1027.081	2
...	...	...	...	...	...
2010-09-26	2010	9	26	896.103	2
2010-09-27	2010	9	27	881.573	2
2010-09-28	2010	9	28	881.573	2
2010-09-29	2010	9	29	867.188	2
2010-09-30	2010	9	30	852.945	2

10957 rows × 5 columns

	Year	Month	Day	Streamflow	Quality
1980-10-01	1980	10	1	2653.697	2
1980-10-02	1980	10	2	2626.756	2
1980-10-03	1980	10	3	2599.999	2
1980-10-04	1980	10	4	2560.203	2
1980-10-05	1980	10	5	2599.999	2
...	...	...	...	...	...
2010-09-26	2010	9	26	2367.308	2
2010-09-27	2010	9	27	2305.231	2
2010-09-28	2010	9	28	2244.249	2
2010-09-29	2010	9	29	2196.246	2
2010-09-30	2010	9	30	2172.502	2

10957 rows × 5 columns

Obidos_record_monthly = Obidos_record['Streamflow'].resample('M').mean()
Obidos_record_monthly = Obidos_record_monthly.to_xarray()
Obidos_record_annual = Obidos_record_monthly.groupby("index.year").max(dim="index")

Xingu_record_monthly = Xingu_record['Streamflow'].resample('M').mean()
Xingu_record_monthly = Xingu_record_monthly.to_xarray()
Xingu_record_annual = Xingu_record_monthly.groupby("index.year").max(dim="index")

Tapajos_record_monthly = Tapajos_record['Streamflow'].resample('M').mean()
Tapajos_record_monthly = Tapajos_record_monthly.to_xarray()
Tapajos_record_annual = Tapajos_record_monthly.groupby("index.year").max(dim="index")

We want observations representative of streamflow at the mouth of the Amazon. Obidos is the observation station closest to the outlet. The rivers Xingu and Tapajos still join the Amazon river downstream of Obidos. We therefore pool Obidos, Xingu and Tapajos together to represent total streamflow at the river mouth. We sum the monthly streamflow values:

Observations_pooled_monthly = Obidos_record_monthly + Xingu_record_monthly + Tapajos_record_monthly
Observations_pooled_annual = Observations_pooled_monthly.groupby("index.year").max(dim="index")

Let’s store these annual maxima streamflow values. We remove the values for the year 1980 because the series starts in October 1980 and the annual maximum flood peak (somewhere in March-May) was not recorded.

Obidos_record_annual_df = Obidos_record_annual.to_dataframe()
Obidos_record_annual_df[1:].to_csv('../Data/Obidos_CABra_record_annual_df.csv')

Xingu_record_annual_df = Xingu_record_annual.to_dataframe()
Xingu_record_annual_df[1:].to_csv('../Data/Xingu_CABra_record_annual_df.csv')

Tapajos_record_annual_df = Tapajos_record_annual.to_dataframe()
Tapajos_record_annual_df[1:].to_csv('../Data/Tapajos_CABra_record_annual_df.csv')

Observations_pooled_annual_df = Observations_pooled_annual.to_dataframe()
Observations_pooled_annual_df[1:].to_csv('../Data/Observations_pooled_annual_df.csv')

Observations_pooled_monthly.to_netcdf('../Data/Observations_pooled_monthly.nc')

Show observed streamflow timeseries¶

Some functions to create the figure for publication:

plt.rcParams["font.family"] = "sans-serif" ##change font
plt.rcParams['font.size'] = 7  ## change font size
# plt.rcParams['svg.fonttype'] = 'none' ## so inkscape recognized texts in svg file
plt.rcParams['pdf.fonttype'] = 42 ## so illustrator can recognize text

plt.figure(figsize=(80/25.4, 60/25.4))
(Observations_pooled_monthly/1000).plot(label = 'Pooled', color = 'blue', linestyle = '--', alpha=0.7)
(Obidos_record['Streamflow']/1000).plot(label = 'Obidos', color = 'brown')
(Xingu_record['Streamflow']/1000).plot(label = 'Xingu', color = 'black')
(Tapajos_record['Streamflow']/1000).plot(label = 'Tapajos', color = 'grey')
plt.legend()
plt.ylabel('Discharge [1000 m3/s] ')
plt.xlabel('')
# plt.savefig('../Graphs/Discharge_subcatchments.pdf',
#             bbox_inches='tight', 
#             dpi =300)
# plt.xlim('2008', '2010')

<Figure size 226.772x170.079 with 0 Axes>

[<matplotlib.lines.Line2D at 0x2ae87c931d90>]

<matplotlib.axes._subplots.AxesSubplot at 0x2ae87bedd880>

<matplotlib.axes._subplots.AxesSubplot at 0x2ae87bedd880>

<matplotlib.axes._subplots.AxesSubplot at 0x2ae87bedd880>

<matplotlib.legend.Legend at 0x2ae87c98bbb0>

Text(0, 0.5, 'Discharge [1000 m3/s] ')

Text(0.5, 0, '')

plt.figure(figsize=(90/25.4, 40/25.4))
(Observations_pooled_monthly.to_dataframe()['Streamflow']/1000).plot(label = 'Pooled', marker='o',
                                                markersize = 3,color = 'blue')
(Obidos_record_monthly.to_dataframe()['Streamflow']/1000).plot(label = 'Obidos',marker='o',
                                                markersize = 3, color = 'brown')
(Xingu_record_monthly.to_dataframe()['Streamflow']/1000).plot(label = 'Xingu',marker='o',
                                                markersize = 3, color = 'black')
(Tapajos_record_monthly.to_dataframe()['Streamflow']/1000).plot(label = 'Tapajos',marker='o',
                                                markersize = 3, color = 'grey')
plt.legend()
plt.ylabel('Discharge [1000 m3/s] ')
plt.xlabel('')
plt.xlim('2008', '2010')
# plt.savefig('../Graphs/Discharge_subcatchments_2009.pdf',
#             bbox_inches='tight', 
#             dpi =300)

<Figure size 255.118x113.386 with 0 Axes>

<matplotlib.axes._subplots.AxesSubplot at 0x2ae87ee641f0>

<matplotlib.axes._subplots.AxesSubplot at 0x2ae87ee641f0>

<matplotlib.axes._subplots.AxesSubplot at 0x2ae87ee641f0>

<matplotlib.axes._subplots.AxesSubplot at 0x2ae87ee641f0>

<matplotlib.legend.Legend at 0x2ae87eedd0d0>

Text(0, 0.5, 'Discharge [1000 m3/s] ')

Text(0.5, 0, '')

(456.0, 480.0)

UNSEEN floods¶

Load the simulations (discharge_monthAvg_presentDay.nc) available at https://doi.org/10.5281/zenodo.4585400 or download them in the ../Data directory by uncommenting and running the following line:

!wget -O ../Data/discharge_monthAvg_presentDay.nc https://zenodo.org/record/4585400/files/discharge_monthAvg_presentDay.nc

--2021-10-07 13:09:36--  https://zenodo.org/record/4585400/files/discharge_monthAvg_presentDay.nc
Resolving zenodo.org (zenodo.org)... 137.138.76.77
Connecting to zenodo.org (zenodo.org)|137.138.76.77|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 247755472 (236M) [application/octet-stream]
Saving to: ‘../Data/discharge_monthAvg_presentDay.nc’

100%[======================================>] 247,755,472 40.7MB/s   in 5.8s   

2021-10-07 13:09:42 (40.7 MB/s) - ‘../Data/discharge_monthAvg_presentDay.nc’ saved [247755472/247755472]

And select grid cells corresponding to (sub)catchment outlets

Amazon = Amazon_simulated['discharge'].sel(lon=-51.75,lat=-1.25)#.sel(lon=-50.25,lat=0.25)  ## Select the timeseries in the gridcell at the mouth of the river
Manaus = Amazon_simulated['discharge'].sel(lon=-59.75,lat=-3.25)  
Obidos = Amazon_simulated['discharge'].sel(lon=-55.75,lat=-2.25)  
Tapajos = Amazon_simulated['discharge'].sel(lon=-55.25,lat=-2.75)  
Xingu = Amazon_simulated['discharge'].sel(lon=-52.25,lat=-1.75)

Now we will select the annual monthly streamflow maxima.

I couldnt assign 2000 years of data to the time dimension of xarray. Hence, I assigned a new coordinate with the years 0-1999.

Store the datasets to be used in R¶

Amazon_simulated_annual_df = Amazon_annual.drop(['lat','lon']).to_dataframe()
Amazon_simulated_annual_df.head()
Amazon_simulated_annual_df.to_csv('../Data/Amazon_annual_df.csv')

Obidos_simulated_annual_df = Obidos_annual.drop(['lat','lon']).to_dataframe()
Obidos_simulated_annual_df.head()
Obidos_simulated_annual_df.to_csv('../Data/Obidos_annual_df.csv')

	discharge
year
0	287012.71875
1	301987.78125
2	358596.53125
3	349216.28125
4	295950.96875

	discharge
year
0	237567.187500
1	241490.000000
2	267892.093750
3	286408.937500
4	234192.484375

Unseen or unrealistic?

Preprocess¶

Observations¶

Show observed streamflow timeseries¶

Longer Obidos record¶

UNSEEN floods¶

Store the datasets to be used in R¶