Process LEMI data from Magdelena Mountains:¶

These data were downloaded in an already-built mth5 format from http://ftp.iris.washington.edu/pub/dropoff/buffer/MT_test_data/time_series/

There is also a prototype LEMI data creation notebook in

mth5/docs/examples/notebooks/lemi_reader_magdelena_2.ipynb

and in the shortcourse :

notebooks/mth5/06_make_mth5_from_lemi.ipynb notebooks/mth5/06a_make_mth5_from_lemi.ipynb

This notebook is set up to use the output from notebooks/mth5/06_make_mth5_from_lemi.ipynb (not 6a).

# Required imports for the program. 
import numpy as np
from pathlib import Path
import pandas as pd
import shutil
import warnings

from mth5.clients.make_mth5 import MakeMTH5
from mth5 import mth5, timeseries
from mth5.utils.helpers import initialize_mth5
from mt_metadata.utils.mttime import get_now_utc, MTime
from aurora.config import BANDS_DEFAULT_FILE
from aurora.config.config_creator import ConfigCreator
from aurora.pipelines.process_mth5 import process_mth5
from mtpy.processing import KernelDataset, RunSummary

warnings.filterwarnings('ignore')

Set up the path to the file¶

here = Path(".").absolute()
mth5_path = here.parent.joinpath("mth5", "from_lemi_02.h5")
assert(mth5_path.exists())

# Sample code for moving files from shared drive

# file_base = "from_lemi.h5"
# source_folder = Path().home().joinpath("shared", "shortcourses", "mt", "lemi")
# source_file = source_folder.joinpath(file_base)
# assert(source_file.exists())

# here = Path(".").absolute()
# target_folder = here.parent.parent.joinpath("data", "timeseries", "lemi")
# target_folder.mkdir(exist_ok=True, parents=True)
# destination_file = target_folder.joinpath(file_base)

# shutil.copyfile(source_file, destination_file)

# mth5_path = destination_file


#mth5_path.exists()

True

mth5_object = initialize_mth5(mth5_path)

# Print some info about the mth5 
mth5_filename = mth5_object.filename
version = mth5_object.file_version
print(f" Filename: {mth5_filename} \n Version: {version}")

 Filename: /home/kkappler/software/irismt/earthscope-mt-course/data/time_series/lemi/from_lemi.h5 
 Version: 0.2.0

#mth5_object.close_mth5()

channel_summary_df = mth5_object.channel_summary.to_dataframe()
channel_summary_df

mth5_object.channel_summary.clear_table()
channel_summary = mth5_object.channel_summary.summarize()
channel_summary_df = mth5_object.channel_summary.to_dataframe()

channel_summary_df

Compress the Channel Summary to a RunSummary¶

mth5_run_summary = RunSummary()
mth5_run_summary.from_mth5s([mth5_path,])
run_summary = mth5_run_summary.clone()

24:10:14T13:59:54 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing /home/kkappler/software/irismt/earthscope-mt-course/data/time_series/lemi/from_lemi.h5

run_summary.df

run_summary.mini_summary

local_station_id = "mt001"
remote_station_id = None

kernel_dataset = KernelDataset()
kernel_dataset.from_run_summary(run_summary, local_station_id, remote_station_id)
kernel_dataset.mini_summary

Note how only the overlapping portions of the runs are kept

cc = ConfigCreator()
config = cc.create_from_kernel_dataset(kernel_dataset)
config.channel_nomenclature.keyword = "LEMI12"

24:10:14T13:59:54 | INFO | line:108 |aurora.config.config_creator | determine_band_specification_style | Bands not defined; setting to EMTF BANDS_DEFAULT_FILE

for decimation in config.decimations:
    decimation.window.type = "hamming"

show_plot = True
tf_cls = process_mth5(config,
                    kernel_dataset,
                    units="MT",
                    show_plot=show_plot,
                    z_file_path=None,
                )

Fetching long content....

24:10:14T14:00:02 | INFO | line:124 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 1
24:10:14T14:00:02 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 1 Successfully
24:10:14T14:00:02 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel bx with empty filters list detected
24:10:14T14:00:02 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:02 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:02 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel by with empty filters list detected
24:10:14T14:00:02 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:02 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:02 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel bz with empty filters list detected
24:10:14T14:00:02 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:02 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:02 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel e1 with empty filters list detected
24:10:14T14:00:02 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:02 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel e2 with empty filters list detected
24:10:14T14:00:03 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:03 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel temperature_e with empty filters list detected
24:10:14T14:00:03 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:03 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel temperature_h with empty filters list detected
24:10:14T14:00:03 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:03 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T14:00:03 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel bx with empty filters list detected
24:10:14T14:00:03 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:03 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel by with empty filters list detected
24:10:14T14:00:03 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:03 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel bz with empty filters list detected
24:10:14T14:00:03 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:03 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel e1 with empty filters list detected
24:10:14T14:00:03 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:03 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel e2 with empty filters list detected
24:10:14T14:00:03 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:03 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel temperature_e with empty filters list detected
24:10:14T14:00:03 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:03 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel temperature_h with empty filters list detected
24:10:14T14:00:03 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:03 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:03 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T14:00:03 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 102.915872s  (0.009717Hz)
24:10:14T14:00:03 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 85.631182s  (0.011678Hz)
24:10:14T14:00:03 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 68.881694s  (0.014518Hz)
24:10:14T14:00:03 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 54.195827s  (0.018452Hz)
24:10:14T14:00:03 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 43.003958s  (0.023254Hz)
24:10:14T14:00:04 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 33.310722s  (0.030020Hz)

24:10:14T14:00:04 | INFO | line:124 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 2
24:10:14T14:00:04 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 2 Successfully
24:10:14T14:00:04 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel bx with empty filters list detected
24:10:14T14:00:04 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:04 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:04 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel by with empty filters list detected
24:10:14T14:00:04 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:04 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:04 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel bz with empty filters list detected
24:10:14T14:00:04 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:04 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:04 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel e1 with empty filters list detected
24:10:14T14:00:04 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:04 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:04 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel e2 with empty filters list detected
24:10:14T14:00:04 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:04 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:04 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel temperature_e with empty filters list detected
24:10:14T14:00:04 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:04 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:04 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel temperature_h with empty filters list detected
24:10:14T14:00:04 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:04 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:04 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T14:00:04 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 411.663489s  (0.002429Hz)
24:10:14T14:00:04 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 342.524727s  (0.002919Hz)
24:10:14T14:00:05 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 275.526776s  (0.003629Hz)
24:10:14T14:00:05 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 216.783308s  (0.004613Hz)
24:10:14T14:00:05 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 172.015831s  (0.005813Hz)
24:10:14T14:00:05 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 133.242890s  (0.007505Hz)

24:10:14T14:00:05 | INFO | line:124 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 3
24:10:14T14:00:05 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 3 Successfully
24:10:14T14:00:05 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel bx with empty filters list detected
24:10:14T14:00:05 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:05 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:05 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel by with empty filters list detected
24:10:14T14:00:05 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:05 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:05 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel bz with empty filters list detected
24:10:14T14:00:05 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:05 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:05 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel e1 with empty filters list detected
24:10:14T14:00:05 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:05 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:05 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel e2 with empty filters list detected
24:10:14T14:00:05 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:05 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:05 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel temperature_e with empty filters list detected
24:10:14T14:00:05 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:05 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:05 | WARNING | line:326 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel temperature_h with empty filters list detected
24:10:14T14:00:05 | WARNING | line:340 |aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
24:10:14T14:00:05 | WARNING | line:296 |mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
24:10:14T14:00:05 | INFO | line:354 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:10:14T14:00:05 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 1514.701336s  (0.000660Hz)
24:10:14T14:00:06 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 1042.488956s  (0.000959Hz)
24:10:14T14:00:06 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 723.371271s  (0.001382Hz)
24:10:14T14:00:06 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 532.971560s  (0.001876Hz)
24:10:14T14:00:06 | INFO | line:35 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 412.837995s  (0.002422Hz)

24:10:14T14:00:06 | INFO | line:777 |mth5.mth5 | close_mth5 | Flushing and closing /home/kkappler/software/irismt/earthscope-mt-course/data/time_series/lemi/from_lemi.h5

aurora

Process CAS04 with Remote Reference

aurora

Accessing Magnetic Field Data from EarthScope Using MTH5