Process CAS04 with Remote Reference - Open-source Python packages for Magnetotellurics

This notebook is a companion to the 2024 JOSS manuscript.

This notebook is shows the workflow for getting data from Earthscope for a few example stations and generating transfer functions using aurora. The data download step is based on condensed version of a tutorial in the mth5 documentation which can be found at: https://github.com/kujaku11/mth5/tree/master/docs/examples/notebooks.

# %matplotlib notebook 
# %matplotlib widget

#Imports

import pandas as pd
import pathlib
import os
#from aurora.sandbox.mth5_channel_summary_helpers import channel_summary_to_make_mth5
#from aurora.config import BANDS_DEFAULT_FILE
from aurora.config.config_creator import ConfigCreator
from aurora.pipelines.process_mth5 import process_mth5
from mth5.mth5 import MTH5
from mth5.clients.make_mth5 import FDSN
from mth5.utils.helpers import initialize_mth5
from mtpy.processing import RunSummary, KernelDataset

import logging, sys
logging.disable(sys.maxsize)
import warnings
warnings.filterwarnings('ignore')

Make MTH5 from IRIS Data Managment Center v0.2.0¶

This example demonstrates how to build an MTH5 from data archived at IRIS, it could work with any MT data stored at an FDSN data center (probably).

We will use the mth5.clients.FDSN class to build the file. There is also second way using the more generic mth5.clients.MakeMTH5 class, which will be highlighted below.

Note: this example assumes that data availability (Network, Station, Channel, Start, End) are all previously known. If you do not know the data that you want to download use IRIS tools to get data availability.

Initialize a MakeMTH5 object¶

Here, we are setting the MTH5 file version to 0.2.0 so that we can have multiple surveys in a single file. Also, setting the client to “IRIS”. Here, we are using obspy.clients tools for the request. Here are the available FDSN clients.

Note: Only the “IRIS” client has been tested.

fdsn_object = FDSN(mth5_version='0.2.0')
fdsn_object.client = "IRIS"

Make the data inquiry as a DataFrame¶

There are a few ways to make the inquiry to request data.

Make a DataFrame by hand. Here we will make a list of entries and then create a DataFrame with the proper column names
You can create a CSV file with a row for each entry. There are some formatting that you need to be aware of. That is the column names and making sure that date-times are YYYY-MM-DDThh:mm:ss

Column Name	Description
network	FDSN Network code (2 letters)
station	FDSN Station code (usually 5 characters)
location	FDSN Location code (typically not used for MT)
channel	FDSN Channel code (3 characters)
start	Start time (YYYY-MM-DDThh:mm:ss) UTC
end	End time (YYYY-MM-DDThh:mm:ss) UTC

In the example below, the stage is set to use two stations: CAS04 and NVR08. Commented out is an example of how to add a third station, REV06

channels = ["LFE", "LFN", "LFZ", "LQE", "LQN"]
CAS04 = ["8P", "CAS04",  '2020-06-02T19:00:00', '2020-07-13T19:00:00'] 
NVR08 = ["8P", "NVR08", '2020-06-02T19:00:00', '2020-07-13T19:00:00']
# REV06 = ["8P", "REV06", '2020-06-02T19:00:00', '2020-07-13T19:00:00']

stations = [CAS04, NVR08,]
# stations.append(REV06)

request_list = []
for entry in stations:
    for channel in channels:
        request_list.append(
            [entry[0], entry[1], "", channel, entry[2], entry[3]]
        )

# Turn list into dataframe
request_df =  pd.DataFrame(request_list, columns=fdsn_object.request_columns) 
request_df

%%time

mth5_filename = fdsn_object.make_mth5_from_fdsn_client(request_df, interact=False)

print(f"Created {mth5_filename}")

24:10:14T18:13:40 | INFO | line:685 |mth5.mth5 | _initialize_file | Initialized MTH5 0.2.0 file /home/kkappler/software/irismt/earthscope-mt-course/notebooks/aurora/8P_CAS04_NVR08.h5 in mode w
24:10:14T18:13:53 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_si_units to a CoefficientFilter.
24:10:14T18:13:53 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_dipole_92.000 to a CoefficientFilter.
24:10:14T18:13:53 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_si_units to a CoefficientFilter.
24:10:14T18:13:53 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_dipole_92.000 to a CoefficientFilter.
24:10:14T18:13:54 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_si_units to a CoefficientFilter.
24:10:14T18:13:54 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_dipole_92.000 to a CoefficientFilter.
24:10:14T18:13:54 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_si_units to a CoefficientFilter.
24:10:14T18:13:54 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_dipole_92.000 to a CoefficientFilter.
24:10:14T18:13:54 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_si_units to a CoefficientFilter.
24:10:14T18:13:54 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_dipole_92.000 to a CoefficientFilter.
24:10:14T18:13:54 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_si_units to a CoefficientFilter.
24:10:14T18:13:54 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_dipole_94.000 to a CoefficientFilter.
24:10:14T18:13:54 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_si_units to a CoefficientFilter.
24:10:14T18:13:54 | INFO | line:133 |mt_metadata.timeseries.filters.obspy_stages | create_filter_from_stage | Converting PoleZerosResponseStage electric_dipole_94.000 to a CoefficientFilter.
24:10:14T18:13:55 | INFO | line:331 |mth5.groups.base | _add_group | RunGroup a already exists, returning existing group.
24:10:14T18:13:56 | WARNING | line:656 |mth5.timeseries.run_ts | validate_metadata | start time of dataset 2020-06-02T19:00:00+00:00 does not match metadata start 2020-06-02T18:41:43+00:00 updating metatdata value to 2020-06-02T19:00:00+00:00
24:10:14T18:13:56 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id a. Setting to ch.run_metadata.id to a
24:10:14T18:13:56 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id a. Setting to ch.run_metadata.id to a
24:10:14T18:13:56 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id a. Setting to ch.run_metadata.id to a
24:10:14T18:13:56 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id a. Setting to ch.run_metadata.id to a
24:10:14T18:13:57 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id a. Setting to ch.run_metadata.id to a
24:10:14T18:13:57 | INFO | line:331 |mth5.groups.base | _add_group | RunGroup b already exists, returning existing group.
24:10:14T18:13:57 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id b. Setting to ch.run_metadata.id to b
24:10:14T18:13:58 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id b. Setting to ch.run_metadata.id to b
24:10:14T18:13:58 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id b. Setting to ch.run_metadata.id to b
24:10:14T18:13:58 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id b. Setting to ch.run_metadata.id to b
24:10:14T18:13:58 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id b. Setting to ch.run_metadata.id to b
24:10:14T18:13:58 | INFO | line:331 |mth5.groups.base | _add_group | RunGroup c already exists, returning existing group.
24:10:14T18:13:59 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id c. Setting to ch.run_metadata.id to c
24:10:14T18:13:59 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id c. Setting to ch.run_metadata.id to c
24:10:14T18:14:00 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id c. Setting to ch.run_metadata.id to c
24:10:14T18:14:00 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id c. Setting to ch.run_metadata.id to c
24:10:14T18:14:00 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id c. Setting to ch.run_metadata.id to c
24:10:14T18:14:00 | INFO | line:331 |mth5.groups.base | _add_group | RunGroup d already exists, returning existing group.
24:10:14T18:14:01 | WARNING | line:669 |mth5.timeseries.run_ts | validate_metadata | end time of dataset 2020-07-13T19:00:00+00:00 does not match metadata end 2020-07-13T21:46:12+00:00 updating metatdata value to 2020-07-13T19:00:00+00:00
24:10:14T18:14:01 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id d. Setting to ch.run_metadata.id to d
24:10:14T18:14:01 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id d. Setting to ch.run_metadata.id to d
24:10:14T18:14:01 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id d. Setting to ch.run_metadata.id to d
24:10:14T18:14:01 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id d. Setting to ch.run_metadata.id to d
24:10:14T18:14:02 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id d. Setting to ch.run_metadata.id to d
24:10:14T18:14:02 | INFO | line:331 |mth5.groups.base | _add_group | RunGroup a already exists, returning existing group.
24:10:14T18:14:02 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id a. Setting to ch.run_metadata.id to a
24:10:14T18:14:02 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id a. Setting to ch.run_metadata.id to a
24:10:14T18:14:02 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id a. Setting to ch.run_metadata.id to a
24:10:14T18:14:03 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id a. Setting to ch.run_metadata.id to a
24:10:14T18:14:03 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id a. Setting to ch.run_metadata.id to a
24:10:14T18:14:03 | INFO | line:331 |mth5.groups.base | _add_group | RunGroup b already exists, returning existing group.
24:10:14T18:14:04 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id b. Setting to ch.run_metadata.id to b
24:10:14T18:14:04 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id b. Setting to ch.run_metadata.id to b
24:10:14T18:14:04 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id b. Setting to ch.run_metadata.id to b
24:10:14T18:14:04 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id b. Setting to ch.run_metadata.id to b
24:10:14T18:14:05 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id b. Setting to ch.run_metadata.id to b
24:10:14T18:14:05 | INFO | line:331 |mth5.groups.base | _add_group | RunGroup c already exists, returning existing group.
24:10:14T18:14:05 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id c. Setting to ch.run_metadata.id to c
24:10:14T18:14:06 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id c. Setting to ch.run_metadata.id to c
24:10:14T18:14:06 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id c. Setting to ch.run_metadata.id to c
24:10:14T18:14:06 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id c. Setting to ch.run_metadata.id to c
24:10:14T18:14:06 | WARNING | line:677 |mth5.groups.run | from_runts | Channel run.id sr1_001 !=  group run.id c. Setting to ch.run_metadata.id to c
24:10:14T18:14:06 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing /home/kkappler/software/irismt/earthscope-mt-course/notebooks/aurora/8P_CAS04_NVR08.h5
24:10:14T18:14:06 | WARNING | line:330 |mth5.mth5 | filename | MTH5 file is not open or has not been created yet. Returning default name
Created /home/kkappler/software/irismt/earthscope-mt-course/notebooks/aurora/8P_CAS04_NVR08.h5
CPU times: user 13.8 s, sys: 313 ms, total: 14.1 s
Wall time: 26.5 s

mth5_path = pathlib.Path("8P_CAS04_NVR08.h5")

m = initialize_mth5(mth5_path)
m.channel_summary.summarize()
df = m.channel_summary.to_dataframe()
df

Fix Survey Name¶

The survey name is extracted from the metadata provided by Earthscope
The value of the survey name was changed from “CONUS South” to “CONUS SoCal”, and the notebook was updated to reflect this, however, as of June 30, 2024, the name seems to have changed back to “CONUS South”.
To avoid problems wiht these change of nomencalature we extract the survey name as a variable

survey_id = df["survey"].unique()[0]
survey_id

'CONUS South'

df.station.unique()

array(['CAS04', 'NVR08'], dtype=object)

mth5_run_summary = RunSummary()
mth5_run_summary.from_mth5s([mth5_path,])
run_summary = mth5_run_summary.clone()
run_summary.df

coverage_short_list_columns = ["survey", 'station', 'run', 'start', 'end', ]
run_summary.df[coverage_short_list_columns]

kernel_dataset = KernelDataset()
station_id = "CAS04"
remote_reference_id = "NVR08"
kernel_dataset.from_run_summary(run_summary, station_id, remote_reference_id)
kernel_dataset.mini_summary

kernel_dataset = KernelDataset()
kernel_dataset.from_run_summary(run_summary, station_id, remote_reference_id)
cutoff_duration_in_seconds = 180000
kernel_dataset.drop_runs_shorter_than(cutoff_duration_in_seconds)
kernel_dataset.df[coverage_short_list_columns]

cc = ConfigCreator()
config = cc.create_from_kernel_dataset(kernel_dataset,) 
#                                       emtf_band_file=BANDS_DEFAULT_FILE,)

24:10:14T18:14:07 | INFO | line:108 |aurora.config.config_creator | determine_band_specification_style | Bands not defined; setting to EMTF BANDS_DEFAULT_FILE

config

for dec_level in config.decimations:
    dec_level.window.type = "hamming"

tf_file_base = f"{station_id}_RR{remote_reference_id}"

show_plot = True
z_file_path = pathlib.Path(f"{tf_file_base}.zrr")
tf_cls = process_mth5(config,
                    kernel_dataset,
                    units="MT",
                    show_plot=show_plot,
                    z_file_path=z_file_path,
                )

24:10:14T18:14:07 | INFO | line:277 |aurora.pipelines.transfer_function_kernel | show_processing_summary | Processing Summary Dataframe:
24:10:14T18:14:07 | INFO | line:278 |aurora.pipelines.transfer_function_kernel | show_processing_summary | 
    duration  has_data  n_samples run station       survey       run_hdf5_reference   station_hdf5_reference    fc  remote  stft mth5_obj dec_level  dec_factor  sample_rate  window_duration  num_samples_window  num_samples  num_stft_windows
0   769090.0      True     847649   b   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         0         1.0     1.000000            128.0                 128     769090.0            8011.0
1   769090.0      True     847649   b   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         1         4.0     0.250000            512.0                 128     192272.0            2002.0
2   769090.0      True     847649   b   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         2         4.0     0.062500           2048.0                 128      48068.0             500.0
3   769090.0      True     847649   b   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         3         4.0     0.015625           8192.0                 128      12017.0             124.0
4   856502.0      True    1638043   c   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         0         1.0     1.000000            128.0                 128     856502.0            8921.0
5   856502.0      True    1638043   c   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         1         4.0     0.250000            512.0                 128     214125.0            2230.0
6   856502.0      True    1638043   c   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         2         4.0     0.062500           2048.0                 128      53531.0             557.0
7   856502.0      True    1638043   c   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         3         4.0     0.015625           8192.0                 128      13382.0             139.0
8   769090.0      True     938510   b   NVR08  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         0         1.0     1.000000            128.0                 128     769090.0            8011.0
9   769090.0      True     938510   b   NVR08  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         1         4.0     0.250000            512.0                 128     192272.0            2002.0
10  769090.0      True     938510   b   NVR08  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         2         4.0     0.062500           2048.0                 128      48068.0             500.0
11  769090.0      True     938510   b   NVR08  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         3         4.0     0.015625           8192.0                 128      12017.0             124.0
12  856502.0      True     856503   c   NVR08  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         0         1.0     1.000000            128.0                 128     856502.0            8921.0
13  856502.0      True     856503   c   NVR08  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         1         4.0     0.250000            512.0                 128     214125.0            2230.0
14  856502.0      True     856503   c   NVR08  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         2         4.0     0.062500           2048.0                 128      53531.0             557.0
15  856502.0      True     856503   c   NVR08  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         3         4.0     0.015625           8192.0                 128      13382.0             139.0
24:10:14T18:14:07 | INFO | line:654 |aurora.pipelines.transfer_function_kernel | memory_check | Total memory: 62.74 GB
24:10:14T18:14:07 | INFO | line:658 |aurora.pipelines.transfer_function_kernel | memory_check | Total Bytes of Raw Data: 0.024 GB
24:10:14T18:14:07 | INFO | line:661 |aurora.pipelines.transfer_function_kernel | memory_check | Raw Data will use: 0.039 % of memory
24:10:14T18:14:07 | INFO | line:707 |aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: CONUS South, station: CAS04, run: b-- Fourier coefficients will be computed
24:10:14T18:14:07 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5
24:10:14T18:14:07 | INFO | line:707 |aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: CONUS South, station: CAS04, run: c-- Fourier coefficients will be computed
24:10:14T18:14:07 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5
24:10:14T18:14:07 | INFO | line:707 |aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: CONUS South, station: NVR08, run: b-- Fourier coefficients will be computed
24:10:14T18:14:08 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5
24:10:14T18:14:08 | INFO | line:707 |aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: CONUS South, station: NVR08, run: c-- Fourier coefficients will be computed
24:10:14T18:14:08 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5
24:10:14T18:14:08 | INFO | line:248 |aurora.pipelines.transfer_function_kernel | check_if_fcs_already_exist | FC levels not present
24:10:14T18:14:08 | INFO | line:517 |aurora.pipelines.process_mth5 | process_mth5_legacy | Processing config indicates 4 decimation levels
24:10:14T18:14:08 | INFO | line:445 |aurora.pipelines.transfer_function_kernel | valid_decimations | After validation there are 4 valid decimation levels
24:10:14T18:14:09 | WARNING | line:656 |mth5.timeseries.run_ts | validate_metadata | start time of dataset 2020-06-03T20:14:13+00:00 does not match metadata start 2020-06-02T22:24:55+00:00 updating metatdata value to 2020-06-03T20:14:13+00:00
24:10:14T18:14:10 | WARNING | line:669 |mth5.timeseries.run_ts | validate_metadata | end time of dataset 2020-06-12T17:52:23+00:00 does not match metadata end 2020-06-14T16:56:02+00:00 updating metatdata value to 2020-06-12T17:52:23+00:00
24:10:14T18:14:11 | WARNING | line:656 |mth5.timeseries.run_ts | validate_metadata | start time of dataset 2020-06-14T18:00:44+00:00 does not match metadata start 2020-06-12T18:32:17+00:00 updating metatdata value to 2020-06-14T18:00:44+00:00
24:10:14T18:14:11 | WARNING | line:669 |mth5.timeseries.run_ts | validate_metadata | end time of dataset 2020-06-24T15:55:46+00:00 does not match metadata end 2020-07-01T17:32:59+00:00 updating metatdata value to 2020-06-24T15:55:46+00:00
24:10:14T18:14:12 | INFO | line:900 |mtpy.processing.kernel_dataset | initialize_dataframe_for_processing | Dataset dataframe initialized successfully
24:10:14T18:14:12 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 0 Successfully
24:10:14T18:14:14 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:15 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:16 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:17 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:17 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 25.728968s  (0.038867Hz)
24:10:14T18:14:18 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 19.929573s  (0.050177Hz)
24:10:14T18:14:18 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 15.164131s  (0.065945Hz)
24:10:14T18:14:18 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 11.746086s  (0.085135Hz)
24:10:14T18:14:19 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 9.195791s  (0.108745Hz)
24:10:14T18:14:19 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 7.362526s  (0.135823Hz)
24:10:14T18:14:19 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 5.856115s  (0.170762Hz)
24:10:14T18:14:20 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 4.682492s  (0.213562Hz)

24:10:14T18:14:21 | INFO | line:124 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 1
24:10:14T18:14:21 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 1 Successfully
24:10:14T18:14:22 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:23 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:23 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:24 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:24 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 102.915872s  (0.009717Hz)
24:10:14T18:14:24 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 85.631182s  (0.011678Hz)
24:10:14T18:14:24 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 68.881694s  (0.014518Hz)
24:10:14T18:14:24 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 54.195827s  (0.018452Hz)
24:10:14T18:14:25 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 43.003958s  (0.023254Hz)
24:10:14T18:14:25 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 33.310722s  (0.030020Hz)

24:10:14T18:14:25 | INFO | line:124 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 2
24:10:14T18:14:26 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 2 Successfully
24:10:14T18:14:26 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:26 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:27 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:27 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:27 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 411.663489s  (0.002429Hz)
24:10:14T18:14:27 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 342.524727s  (0.002919Hz)
24:10:14T18:14:27 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 275.526776s  (0.003629Hz)
24:10:14T18:14:27 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 216.783308s  (0.004613Hz)
24:10:14T18:14:27 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 172.015831s  (0.005813Hz)
24:10:14T18:14:27 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 133.242890s  (0.007505Hz)

24:10:14T18:14:28 | INFO | line:124 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 3
24:10:14T18:14:28 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 3 Successfully
24:10:14T18:14:28 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:29 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:29 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:29 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T18:14:29 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 1514.701336s  (0.000660Hz)
24:10:14T18:14:29 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 1042.488956s  (0.000959Hz)
24:10:14T18:14:29 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 723.371271s  (0.001382Hz)
24:10:14T18:14:29 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 532.971560s  (0.001876Hz)
24:10:14T18:14:30 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 412.837995s  (0.002422Hz)

24:10:14T18:14:30 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5
24:10:14T18:14:31 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5

tf_cls.write(fn=f"{tf_file_base}.xml", file_type="emtfxml")
tf_cls.write(fn=f"{tf_file_base}.edi", file_type="edi")
tf_cls.write(fn=f"{tf_file_base}.zrr", file_type="zrr")

MT( station='CAS04', latitude=37.63, longitude=-121.47, elevation=335.26 )

archived_z_file = pathlib.Path(f"CAS04bcd_REV06.zrr")

from aurora.transfer_function.plot.comparison_plots import compare_two_z_files
z_file_path = "CAS04_RRNVR08.zrr"

To compare with the archived file, we need to set the coordinate system to geographic¶

The TF will be output with a header like this:

TRANSFER FUNCTIONS IN MEASUREMENT COORDINATES
********* WITH FULL ERROR COVARIANCE ********
Aurora Robust Remote Reference
station: CAS04
coordinate    37.633  -121.468 declination    13.17
number of channels    5  number of frequencies  25
 orientations and tilts of each channel
    1    13.20     0.00 CAS04  Hx
    2   103.20     0.00 CAS04  Hy
    3     0.00    90.00 CAS04  Hz
    4    13.20     0.00 CAS04  Ex
    5   103.20     0.00 CAS04  Ey

To remove the rotation, we can use a variety of tools, but another way is just to overwrite the orientations:

TRANSFER FUNCTIONS IN MEASUREMENT COORDINATES
********* WITH FULL ERROR COVARIANCE ********
 Aurora Robust Remote Reference
station: CAS04
coordinate    37.633  -121.468 declination    13.17
number of channels    5  number of frequencies  25
 orientations and tilts of each channel
    1     0.00     0.00 CAS04  Hx
    2    90.00     0.00 CAS04  Hy
    3     0.00    90.00 CAS04  Hz
    4     0.00     0.00 CAS04  Ex
    5    90.00     0.00 CAS04  Ey

This is why we set angle1=13.2 degrees in the comparison plotter.

print(z_file_path)
print(archived_z_file)
print(tf_file_base)

CAS04_RRNVR08.zrr
CAS04bcd_REV06.zrr
CAS04_RRNVR08

compare_two_z_files(
    z_file_path,
    archived_z_file,
    angle1=+13.2,
    label1="aurora",
    label2="emtf",
    scale_factor1=1,
    out_file=f"{tf_file_base}compare.png",
    markersize=3,
    rho_ylims=[1e0, 1e3],
    xlims=[0.99, 2000],
    rho_ax_label_size=12,
    phi_ax_label_size=12
)

24:10:14T18:14:31 | INFO | line:86 |aurora.transfer_function.plot.comparison_plots | compare_two_z_files | Sacling TF scale_factor1: 1

Part II: Logic to save FCs¶

Storage of FCs was intended to be an option to provide to users on the fly, by setting the decimation_level part of the processing config to dec_level.save_fcs = True and dec_level.save_fcs_type = "h5".

This works in some cases but not in general. Details are in aurora issue #319 simpeg/aurora#319.

The proposed solution is to generate FCs per station by processing as a single station.

We start with the Run Summary table:

mth5_run_summary = RunSummary()
mth5_run_summary.from_mth5s([mth5_path,])
run_summary = mth5_run_summary.clone()
run_summary.df

But this time, process stations individually (e.g. CAS04 as below)¶

kernel_dataset = KernelDataset()
station_id = "CAS04"
remote_reference_id = None
kernel_dataset.from_run_summary(run_summary, station_id, remote_reference_id)
kernel_dataset.mini_summary

Before adding the FCs, take a look at the file stats:

file_info = os.stat(mth5_path)
print(f"file_info: \n {file_info}")

file_size_before_fc_addition = file_info.st_size
print(f"file_size_before_fc_addition {file_size_before_fc_addition}")

file_info: 
 os.stat_result(st_mode=33204, st_ino=95309503, st_dev=66306, st_nlink=1, st_uid=1001, st_gid=1001, st_size=107289751, st_atime=1728954872, st_mtime=1728954872, st_ctime=1728954872)
file_size_before_fc_addition 107289751

cc = ConfigCreator()
config = cc.create_from_kernel_dataset(kernel_dataset,) 
for dec_level in config.decimations:
    dec_level.window.type = "hamming"
#    dec_level.window.overlap = int(dec_level.window.num_samples/4)
    dec_level.save_fcs = True
    dec_level.save_fcs_type = "h5"

24:10:14T18:14:32 | INFO | line:108 |aurora.config.config_creator | determine_band_specification_style | Bands not defined; setting to EMTF BANDS_DEFAULT_FILE

config

tf_file_base = f"{station_id}_SS"

show_plot = True
z_file_path = pathlib.Path(f"{tf_file_base}.zrr")
tf_cls = process_mth5(config,
                    kernel_dataset,
                    units="MT",
                    show_plot=show_plot,
                    z_file_path=z_file_path,
                )

24:10:14T18:14:32 | INFO | line:277 |aurora.pipelines.transfer_function_kernel | show_processing_summary | Processing Summary Dataframe:
24:10:14T18:14:32 | INFO | line:278 |aurora.pipelines.transfer_function_kernel | show_processing_summary | 
     duration  has_data  n_samples run station       survey       run_hdf5_reference   station_hdf5_reference    fc  remote  stft mth5_obj dec_level  dec_factor  sample_rate  window_duration  num_samples_window  num_samples  num_stft_windows
0     11266.0      True      11267   a   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         0         1.0     1.000000            128.0                 128      11266.0             117.0
1     11266.0      True      11267   a   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         1         4.0     0.250000            512.0                 128       2816.0              29.0
2     11266.0      True      11267   a   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         2         4.0     0.062500           2048.0                 128        704.0               7.0
3     11266.0      True      11267   a   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         3         4.0     0.015625           8192.0                 128        176.0               1.0
4    847648.0      True     847649   b   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         0         1.0     1.000000            128.0                 128     847648.0            8829.0
5    847648.0      True     847649   b   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         1         4.0     0.250000            512.0                 128     211912.0            2207.0
6    847648.0      True     847649   b   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         2         4.0     0.062500           2048.0                 128      52978.0             551.0
7    847648.0      True     847649   b   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         3         4.0     0.015625           8192.0                 128      13244.0             137.0
8   1638042.0      True    1638043   c   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         0         1.0     1.000000            128.0                 128    1638042.0           17062.0
9   1638042.0      True    1638043   c   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         1         4.0     0.250000            512.0                 128     409510.0            4265.0
10  1638042.0      True    1638043   c   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         2         4.0     0.062500           2048.0                 128     102377.0            1066.0
11  1638042.0      True    1638043   c   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         3         4.0     0.015625           8192.0                 128      25594.0             266.0
12  1034585.0      True    1034586   d   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         0         1.0     1.000000            128.0                 128    1034585.0           10776.0
13  1034585.0      True    1034586   d   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         1         4.0     0.250000            512.0                 128     258646.0            2693.0
14  1034585.0      True    1034586   d   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         2         4.0     0.062500           2048.0                 128      64661.0             673.0
15  1034585.0      True    1034586   d   CAS04  CONUS South  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         3         4.0     0.015625           8192.0                 128      16165.0             168.0
24:10:14T18:14:32 | INFO | line:411 |aurora.pipelines.transfer_function_kernel | validate_processing | No RR station specified, switching RME_RR to RME
24:10:14T18:14:32 | INFO | line:411 |aurora.pipelines.transfer_function_kernel | validate_processing | No RR station specified, switching RME_RR to RME
24:10:14T18:14:32 | INFO | line:411 |aurora.pipelines.transfer_function_kernel | validate_processing | No RR station specified, switching RME_RR to RME
24:10:14T18:14:32 | INFO | line:411 |aurora.pipelines.transfer_function_kernel | validate_processing | No RR station specified, switching RME_RR to RME
24:10:14T18:14:32 | INFO | line:654 |aurora.pipelines.transfer_function_kernel | memory_check | Total memory: 62.74 GB
24:10:14T18:14:32 | INFO | line:658 |aurora.pipelines.transfer_function_kernel | memory_check | Total Bytes of Raw Data: 0.026 GB
24:10:14T18:14:32 | INFO | line:661 |aurora.pipelines.transfer_function_kernel | memory_check | Raw Data will use: 0.042 % of memory
24:10:14T18:14:32 | INFO | line:707 |aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: CONUS South, station: CAS04, run: a-- Fourier coefficients will be computed
24:10:14T18:14:33 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5
24:10:14T18:14:33 | INFO | line:707 |aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: CONUS South, station: CAS04, run: b-- Fourier coefficients will be computed
24:10:14T18:14:33 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5
24:10:14T18:14:33 | INFO | line:707 |aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: CONUS South, station: CAS04, run: c-- Fourier coefficients will be computed
24:10:14T18:14:33 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5
24:10:14T18:14:33 | INFO | line:707 |aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: CONUS South, station: CAS04, run: d-- Fourier coefficients will be computed
24:10:14T18:14:33 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5
24:10:14T18:14:33 | INFO | line:248 |aurora.pipelines.transfer_function_kernel | check_if_fcs_already_exist | FC levels not present
24:10:14T18:14:33 | INFO | line:517 |aurora.pipelines.process_mth5 | process_mth5_legacy | Processing config indicates 4 decimation levels
24:10:14T18:14:33 | INFO | line:445 |aurora.pipelines.transfer_function_kernel | valid_decimations | After validation there are 4 valid decimation levels
24:10:14T18:14:38 | INFO | line:900 |mtpy.processing.kernel_dataset | initialize_dataframe_for_processing | Dataset dataframe initialized successfully
24:10:14T18:14:38 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 0 Successfully
24:10:14T18:14:39 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:40 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:42 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:43 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:43 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 25.728968s  (0.038867Hz)
24:10:14T18:14:43 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 19.929573s  (0.050177Hz)
24:10:14T18:14:44 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 15.164131s  (0.065945Hz)
24:10:14T18:14:44 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 11.746086s  (0.085135Hz)
24:10:14T18:14:45 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 9.195791s  (0.108745Hz)
24:10:14T18:14:45 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 7.362526s  (0.135823Hz)
24:10:14T18:14:46 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 5.856115s  (0.170762Hz)
24:10:14T18:14:47 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 4.682492s  (0.213562Hz)

24:10:14T18:14:48 | INFO | line:124 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 1
24:10:14T18:14:49 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 1 Successfully
24:10:14T18:14:49 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:50 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:50 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:51 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:51 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 102.915872s  (0.009717Hz)
24:10:14T18:14:51 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 85.631182s  (0.011678Hz)
24:10:14T18:14:52 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 68.881694s  (0.014518Hz)
24:10:14T18:14:52 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 54.195827s  (0.018452Hz)
24:10:14T18:14:52 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 43.003958s  (0.023254Hz)
24:10:14T18:14:52 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 33.310722s  (0.030020Hz)

24:10:14T18:14:53 | INFO | line:124 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 2
24:10:14T18:14:53 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 2 Successfully
24:10:14T18:14:53 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:54 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:54 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:55 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:55 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 411.663489s  (0.002429Hz)
24:10:14T18:14:55 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 342.524727s  (0.002919Hz)
24:10:14T18:14:55 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 275.526776s  (0.003629Hz)
24:10:14T18:14:55 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 216.783308s  (0.004613Hz)
24:10:14T18:14:56 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 172.015831s  (0.005813Hz)
24:10:14T18:14:56 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 133.242890s  (0.007505Hz)

24:10:14T18:14:56 | INFO | line:124 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 3
24:10:14T18:14:56 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 3 Successfully
24:10:14T18:14:57 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:57 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:58 | INFO | line:364 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Saving FC level
24:10:14T18:14:58 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 1514.701336s  (0.000660Hz)
24:10:14T18:14:58 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 1042.488956s  (0.000959Hz)
24:10:14T18:14:58 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 723.371271s  (0.001382Hz)
24:10:14T18:14:58 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 532.971560s  (0.001876Hz)
24:10:14T18:14:58 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 412.837995s  (0.002422Hz)

24:10:14T18:14:59 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing 8P_CAS04_NVR08.h5

file_info = os.stat(mth5_path)
print(f"file_info: \n {file_info}")

file_size_after_fc_addition = file_info.st_size
print(f"file_size_before_fc_addition {file_size_before_fc_addition}")
print(f"file_size_after_fc_addition {file_size_after_fc_addition}")

file_info: 
 os.stat_result(st_mode=33204, st_ino=95309503, st_dev=66306, st_nlink=1, st_uid=1001, st_gid=1001, st_size=358591655, st_atime=1728954899, st_mtime=1728954899, st_ctime=1728954899)
file_size_before_fc_addition 107289751
file_size_after_fc_addition 358591655

Now that the FCs are saved we can access them:¶

These plats are intended to be put in spectrogram class

# Choose what specific FCs we want:
# survey_id = "CONUS SoCal"  # declared directly from dataframe to avoid spurious name changes in archived metadata
station_id = "CAS04"
run_id = "b"
decimation_level_id = "0"

m = initialize_mth5(mth5_path)

survey_group = m.get_survey(survey_id)

station_obj = survey_group.stations_group.get_station(station_id)

fc_group = station_obj.fourier_coefficients_group.get_fc_group(run_id)
fc_decimation_level = fc_group.get_decimation_level(decimation_level_id)
stft_obj = fc_decimation_level.to_xarray()

stft_obj

import matplotlib.pyplot as plt
import numpy as np

ex = stft_obj.ex
ex = ex.dropna(dim="frequency")
ex

ex = np.abs(ex)

ex.time.data

array(['2020-06-02T22:24:55.000000000', '2020-06-02T22:26:31.000000000',
       '2020-06-02T22:28:07.000000000', ...,
       '2020-06-12T17:46:31.000000000', '2020-06-12T17:48:07.000000000',
       '2020-06-12T17:49:43.000000000'], dtype='datetime64[ns]')

Plotting spectrograms with dates:

The cell below was adapted from:

https://stackoverflow.com/questions/23139595/dates-in-the-xaxis-for-a-matplotlib-plot-with-imshow

import numpy as np
import matplotlib.pyplot as plt

import matplotlib.dates as mdates

import datetime as dt

x_lims = [ex.time.data[0], ex.time.data[-1]]

# You can then convert these datetime.datetime objects to the correct
# format for matplotlib to work with.
x_lims = mdates.date2num(x_lims)

# Set y-limits.
y_lims = [ex.frequency.data[0], ex.frequency.data[-1]]

fig, ax = plt.subplots()

# Using ax.imshow we set two keyword arguments. The first is extent.
# We give extent the values from x_lims and y_lims above.
# We also set the aspect to "auto" which should set the plot up nicely.
ax.imshow(np.log10(ex.T), extent = [x_lims[0], x_lims[1],  y_lims[0], y_lims[1]], 
          aspect='auto', origin='lower' )

# # We tell Matplotlib that the x-axis is filled with datetime data, 
# # this converts it from a float (which is the output of date2num) 
# # into a nice datetime string.
ax.xaxis_date()

# # We can use a DateFormatter to choose how this datetime string will look.
# # I have chosen HH:MM:SS though you could add DD/MM/YY if you had data
# # over different days.
date_format = mdates.DateFormatter('%Y-%m-%d')# %H:%M:%S')

ax.xaxis.set_major_formatter(date_format)

# # This simply sets the x-axis data to diagonal so it fits better.
fig.autofmt_xdate()
ax.set_ylabel("Frequency (Hz)")
ax.set_xlabel("Time")
ax.set_title(f"log_{10} Amplitude Spectrogram for {station_id}, run {run_id}")
plt.show()

Absolute Minimal Example¶

This is the code from Figure 3 in the JOSS manuscript intended to show that the processing can be run in 8 lines including saving results to edi file format.

from aurora.config.config_creator import ConfigCreator
from aurora.pipelines.process_mth5 import process_mth5
from mtpy.processing import KernelDataset, RunSummary

run_summary = RunSummary()
run_summary.from_mth5s(["8P_CAS04_NVR08.h5",])
kernel_dataset = KernelDataset()
kernel_dataset.from_run_summary(run_summary, "CAS04", "NVR08")
cc = ConfigCreator()
config = cc.create_from_kernel_dataset(kernel_dataset) 
tf = process_mth5(config, kernel_dataset)
tf.write(fn="CAS04_rrNVR08.edi", file_type="edi")

Fetching long content....

Station: CAS04
--------------------------------------------------
	Survey:        CONUS South
	Project:       USMTArray
	Acquired by:   None
	Acquired date: 2020-06-02
	Latitude:      37.633
	Longitude:     -121.468
	Elevation:     335.262
	Impedance:     True
	Tipper:        True
	Number of periods: 25
		Period Range:   4.68249E+00 -- 1.51470E+03 s
		Frequency Range 6.60196E-04 -- 2.13561E-01 s

aurora

Processing Configuration

aurora

Process LEMI data from Magdelena Mountains: