Getting Started

Emanuel Rodriguez

2018-09-10

Install

You can install CDECRetrieve the usual way,

# for stable version
install.packages("CDECRetrieve")

# for development version
devtools::install_github("flowwest/CDECRetrieve")

Intro

The goal for CDECRetrieve is to create a workflow for R users using CDEC data, we believe that a well defined workflow is easier to automate and less prone to error (or easier to catch errors). In order to do this we create “services” out of different endpoints available through the CDEC site. A lot ideas in developing the package came from using dataRetrieval from USGS and the NOAA CDO api.

Exploring Locations

We start by first exploring locations of interest. The CDEC site provides a web form with a lot of options,

cdec station search

cdec station search

The pakcage exposes this functionallity through cdec_stations(). Although it doesn’t (currently) map all options in the web form it does so for the most used, namely, station id, nearby city, river basin, hydro area and county. At least one of the parameters must be supplied, and combination of these can be supplied to refine the search.

library(CDECRetrieve)

cdec_stations(station_id = "kwk") # return metadata for KWK
#> # A tibble: 0 x 9
#> # ... with 9 variables: station_id <chr>, name <chr>, river_basin <chr>,
#> #   county <chr>, longitude <lgl>, latitude <lgl>, elevation <lgl>,
#> #   operator <lgl>, state <chr>

# show all locations near san francisco, this returns a set of 
# CDEC station that are near San Francisco
cdec_stations(nearby_city = "san francisco")
#> # A tibble: 3 x 9
#>   station_id name  river_basin county longitude latitude elevation operator
#>   <chr>      <chr> <chr>       <chr>      <dbl>    <dbl>     <int> <chr>   
#> 1 cx2        dail… sf bay      san f…     -122.     37.8         0 CA Dept…
#> 2 ggt        gold… sf bay      san f…     -122.     37.8         0 Nationa…
#> 3 sfn        san … sf bay      san f…     -122.     37.8       150 Nationa…
#> # ... with 1 more variable: state <chr>

# show all location in the sf bay river basin
cdec_stations(river_basin = "sf bay")
#> # A tibble: 24 x 9
#>    station_id name  river_basin county longitude latitude elevation
#>    <chr>      <chr> <chr>       <chr>      <dbl>    <dbl> <chr>    
#>  1 lfy        lafa… sf bay      contr…     -122.     37.9 465      
#>  2 mas        main… sf bay      santa…    -1000.    100.0 9,999    
#>  3 spb        san … sf bay      contr…     -122.     37.9 330      
#>  4 ttk        term… sf bay      santa…     -122.     37.4 650      
#>  5 mrh        mars… sf bay      contr…     -122.     37.9 740      
#>  6 cx2        dail… sf bay      san f…     -122.     37.8 0        
#>  7 acm        arro… sf bay      marin      -123.     37.9 3        
#>  8 paa        palo… sf bay      santa…     -122.     37.4 7        
#>  9 sww        swee… san antoni… santa…     -121.     37.4 2,150    
#> 10 ggt        gold… sf bay      san f…     -122.     37.8 0        
#> # ... with 14 more rows, and 2 more variables: operator <chr>, state <chr>

# show all station in Tehama county
cdec_stations(county = "tehama")
#> # A tibble: 45 x 9
#>    station_id name  river_basin county longitude latitude elevation
#>    <chr>      <chr> <chr>       <chr>      <dbl>    <dbl> <chr>    
#>  1 blb        blac… stony cr    tehama     -122.     39.8 426      
#>  2 rbf        red … sacto vly … tehama     -122.     40.2 353      
#>  3 sh1        shee… sacramento… tehama     -123.     39.5 6,500    
#>  4 cot        cott… cottonwood… tehama     -122.     40.4 364      
#>  5 ctn        cott… cottonwood… tehama     -123.     40.3 3,400    
#>  6 mlm        mill… sacramento… tehama     -122.     40.1 385      
#>  7 mnr        mine… sacto vly … tehama     -122.     40.4 4,875    
#>  8 sbb        sacr… sacto vly … tehama     -122.     40.3 186      
#>  9 vin        sacr… sacramento… tehama     -122.     39.9 185      
#> 10 crg        corn… sacramento… tehama     -122.     39.9 294      
#> # ... with 35 more rows, and 2 more variables: operator <chr>, state <chr>

Since we are simply exploring for locations of interest, it may be useful to map these for visual inspection. CDECRetrieve provides a simple function to do exactly this map_stations().

library(magrittr)
library(leaflet)

cdec_stations(county = "tehama") %>% 
  map_stations()

The same can be done with leaflet functions

d <- cdec_stations(county = "tehama")
leaflet(d) %>% 
  addTiles() %>% 
  addCircleMarkers(label=~station_id) #psk is way off here 

Exploring Datasets within a Station

After exploring stations in a desired location. We can start focusing on the datasets available at the locations.

station <- "sha"
cdec_datasets("sha")
#> # A tibble: 21 x 6
#>    sensor_number sensor_name   sensor_units duration start      end       
#>            <int> <chr>         <chr>        <chr>    <date>     <date>    
#>  1             2 precipitatio… inches       daily    2003-10-01 2018-09-10
#>  2             2 precipitatio… inches       monthly  1953-10-01 2018-09-10
#>  3             6 reservoir el… feet         daily    1985-01-01 2018-09-10
#>  4             6 reservoir el… feet         hourly   1993-12-09 2018-09-10
#>  5             8 full natural… cfs          daily    1987-05-31 2018-09-10
#>  6            15 reservoir st… af           daily    1985-01-01 2018-09-10
#>  7            15 reservoir st… af           hourly   1994-06-24 2018-09-10
#>  8            15 reservoir st… af           monthly  1953-10-01 2018-09-10
#>  9            22 reservoir st… af           daily    1993-10-03 2018-09-10
#> 10            23 reservoir ou… cfs          daily    1987-01-05 2018-09-10
#> # ... with 11 more rows

Since all of these functions return a tidy dataframe we can make use of the dplyr to filter, mutate and explore. Here we look for datasets in Shasta that report a storage

library(magrittr)

cdec_datasets("sha") %>% 
  dplyr::filter(grepl("storage", sensor_name))
#> # A tibble: 5 x 6
#>   sensor_number sensor_name    sensor_units duration start      end       
#>           <int> <chr>          <chr>        <chr>    <date>     <date>    
#> 1            15 reservoir sto… af           daily    1985-01-01 2018-09-10
#> 2            15 reservoir sto… af           hourly   1994-06-24 2018-09-10
#> 3            15 reservoir sto… af           monthly  1953-10-01 2018-09-10
#> 4            22 reservoir sto… af           daily    1993-10-03 2018-09-10
#> 5            94 reservoir top… af           daily    2000-10-24 2018-09-10

Take note of the sensor number, and duration, these will be needed for querying data in the next section.

Query Data

Now that we have a location, parameter of interest and duration we can start to query for actual data.

sha_storage_daily <- cdec_query(station = "sha", sensor_num = "15", 
                                dur_code = "d", start_date = "2018-01-01", 
                                end_date = Sys.Date())

sha_storage_daily
#> # A tibble: 253 x 5
#>    agency_cd location_id datetime            parameter_cd parameter_value
#>    <chr>     <chr>       <dttm>              <chr>                  <dbl>
#>  1 CDEC      SHA         2018-01-01 00:00:00 15                   3203249
#>  2 CDEC      SHA         2018-01-02 00:00:00 15                   3202064
#>  3 CDEC      SHA         2018-01-03 00:00:00 15                   3203723
#>  4 CDEC      SHA         2018-01-04 00:00:00 15                   3206566
#>  5 CDEC      SHA         2018-01-05 00:00:00 15                   3210358
#>  6 CDEC      SHA         2018-01-06 00:00:00 15                   3215097
#>  7 CDEC      SHA         2018-01-07 00:00:00 15                   3217003
#>  8 CDEC      SHA         2018-01-08 00:00:00 15                   3229391
#>  9 CDEC      SHA         2018-01-09 00:00:00 15                   3237014
#> 10 CDEC      SHA         2018-01-10 00:00:00 15                   3242032
#> # ... with 243 more rows

Once again the the data is in a tidy form.

Plot

We can plot with ggplot2

library(ggplot2)

sha_storage_daily %>% 
  ggplot(aes(datetime, parameter_value)) + geom_line()