Skip to article frontmatterSkip to article content

Here we will look at different affects of noise.

import numpy as np
from mth5.data.make_mth5_from_asc import create_test12rr_h5
from mth5.data.paths import SyntheticTestPaths
from mth5.mth5 import MTH5

from mtpy.processing import AuroraProcessing

synthetic_test_paths = SyntheticTestPaths()
target_folder = synthetic_test_paths.mth5_path
mth5_path = target_folder.joinpath("test12rr.h5")
if not mth5_path.exists():
    create_test12rr_h5(target_folder=target_folder)  


2024-10-17T11:16:34.501966-0700 | INFO | mth5.mth5 | _initialize_file | Initialized MTH5 0.1.0 file C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\mth5\data\mth5\test12rr.h5 in mode w
2024-10-17T11:16:37.070816-0700 | INFO | mth5.mth5 | close_mth5 | Flushing and closing C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\mth5\data\mth5\test12rr.h5

Noise

We can define some noise methods.

def get_ac_noise(time, amplitude, period):
    """
    Create a sinusoidal noise signal

    time: array of time sampled at a single sample rate
    amplitude: amplitude of the sinusoidal signal
    period: period of the sinusoid in seconds
    """
    return amplitude * np.sin(2 * np.pi * (1 / period) * time)
def get_impulses(time, max_amplitude, n_impulses=50):
    """
    Create a random series of impulses

    time: array of time sampled at a single sample rate
    max_amplitude: amplitude of the impulses
    n_impulses: number of impulses to add
    """
    imp = np.zeros_like(time)
    for ii in range(n_impulses):
        index = np.random.randint(0, imp.size)
        amplitude = (-1)**(np.random.randint(0, 9))*np.random.randint(1, max_amplitude)
        imp[index] = amplitude
    return imp
def get_periodic_noise_electric(time, period=4, repeat=16, max_amplitude=8000):
    """
    Create a series of periodic box cars often seen in electric channels

    time: array of time sampled at a single sample rate
    period: the length of the box car
    repeat: number of samples the box car is repeated
    max_amplitude: maximum amplitude of the box car
    """
    comb = np.zeros_like(time)
    for ii in np.linspace(0, comb.size-(period + 1), repeat):
        comb[int(ii): int(ii)+period] = 1 * max_amplitude
    return comb 
def get_periodic_noise_magnetic(time, period=4, repeat=16, max_amplitude=8000):
    """
    Create a series of periodic box cars responses often seen in magnetic channels

    time: array of time sampled at a single sample rate
    period: the length of the box car
    repeat: number of samples the box car is repeated
    max_amplitude: maximum amplitude of the box car
    """
    comb = np.zeros_like(time)
    for ii in np.linspace(0, comb.size-(period + 1), repeat):
        comb[int(ii)] = 1 * max_amplitude
        comb[int(ii)+period] = -1 * max_amplitude

    gauss = np.exp(np.arange(period)**2 / repeat)
    gauss = gauss / gauss.max() 
    response = max_amplitude * gauss
    comb = np.convolve(comb, response, mode="same")
    return comb 
with MTH5() as m:
    m = m.open_mth5(mth5_path)
    runts = m.get_run("test1", "001").to_runts()
    t = np.arange(len(runts.dataset.time))

    # add AC noise
    new_run = runts.copy()
    new_run.run_metadata.id = "12s_sinusoid"
    for channel in new_run.channels:
        ch = getattr(new_run, channel)
        ac_noise = get_ac_noise(t, 2*ch.ts.max(), 12)
        ch.ts = ch.ts + ac_noise
        setattr(new_run, channel, ch)
    rg = m.add_run("test1", new_run.run_metadata.id)
    rg.from_runts(new_run)

    new_run.plot_spectra()

    # add impulse noise
    new_run = runts.copy()
    new_run.run_metadata.id = "impulses"
    for channel in new_run.channels:
        ch = getattr(new_run, channel)
        impulse_noise = get_impulses(t, 2*ch.ts.max(), 100)
        ch.ts = ch.ts + impulse_noise
        setattr(new_run, channel, ch)
    rg = m.add_run("test1", new_run.run_metadata.id)
    rg.from_runts(new_run)

    new_run.plot_spectra()

    # add periodic noise
    new_run = runts.copy()
    new_run.run_metadata.id = "periodic"
    for channel in new_run.channels:
        ch = getattr(new_run, channel)
        if ch.component in ["ex", "ey"]:
            periodic_noise = get_periodic_noise_electric(t, max_amplitude=1.5*ch.ts.max())
        elif ch.component in ["hx", "hy", "hz"]:
            periodic_noise = get_periodic_noise_magnetic(t, max_amplitude=1.5*ch.ts.max())*1E-3
        ch.ts = ch.ts + periodic_noise
        setattr(new_run, channel, ch)
    rg = m.add_run("test1", new_run.run_metadata.id)
    rg.from_runts(new_run)

    new_run.plot_spectra()

    # Cut Electric line
    new_run = runts.copy()
    new_run.run_metadata.id = "cut_eline"
    bad_ex = new_run.ex.copy()
    bad_ex.ts = np.random.rand(len(t)) * new_run.ex.ts.max() * 0.01
    new_run.ex = bad_ex
    rg = m.add_run("test1", new_run.run_metadata.id)
    rg.from_runts(new_run)

    new_run.plot_spectra()

    # Flipped Horizontal Hx and vertical Hz
    new_run = runts.copy()
    new_run.run_metadata.id = "flipped_hx_hz"
    flipped_hx = new_run.hx.copy()
    flipped_hx.ts = flipped_hx.ts * -1
    new_run.hx = flipped_hx
    flipped_hz = new_run.hz.copy()
    flipped_hz.ts = flipped_hz.ts * -1
    new_run.hz = flipped_hz

    rg = m.add_run("test1", new_run.run_metadata.id)
    rg.from_runts(new_run)

    new_run.plot_spectra()


2024-10-17T11:17:36.970288-0700 | INFO | mth5.groups.base | _add_group | RunGroup 12s_sinusoid already exists, returning existing group.
2024-10-17T11:17:37.089612-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id 12s_sinusoid. Setting to ch.run_metadata.id to 12s_sinusoid
2024-10-17T11:17:37.220440-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id 12s_sinusoid. Setting to ch.run_metadata.id to 12s_sinusoid
2024-10-17T11:17:37.370480-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id 12s_sinusoid. Setting to ch.run_metadata.id to 12s_sinusoid
2024-10-17T11:17:37.504051-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id 12s_sinusoid. Setting to ch.run_metadata.id to 12s_sinusoid
2024-10-17T11:17:37.637481-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id 12s_sinusoid. Setting to ch.run_metadata.id to 12s_sinusoid
<Figure size 640x480 with 2 Axes>
2024-10-17T11:17:39.639166-0700 | INFO | mth5.groups.base | _add_group | RunGroup impulses already exists, returning existing group.
2024-10-17T11:17:39.739221-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id impulses. Setting to ch.run_metadata.id to impulses
2024-10-17T11:17:39.891580-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id impulses. Setting to ch.run_metadata.id to impulses
2024-10-17T11:17:40.022783-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id impulses. Setting to ch.run_metadata.id to impulses
2024-10-17T11:17:40.156221-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id impulses. Setting to ch.run_metadata.id to impulses
2024-10-17T11:17:40.308789-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id impulses. Setting to ch.run_metadata.id to impulses
<Figure size 640x480 with 2 Axes>
2024-10-17T11:17:42.391404-0700 | INFO | mth5.groups.base | _add_group | RunGroup periodic already exists, returning existing group.
2024-10-17T11:17:42.508200-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id periodic. Setting to ch.run_metadata.id to periodic
2024-10-17T11:17:42.660515-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id periodic. Setting to ch.run_metadata.id to periodic
2024-10-17T11:17:42.791705-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id periodic. Setting to ch.run_metadata.id to periodic
2024-10-17T11:17:42.941817-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id periodic. Setting to ch.run_metadata.id to periodic
2024-10-17T11:17:43.077516-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id periodic. Setting to ch.run_metadata.id to periodic
<Figure size 640x480 with 2 Axes>
2024-10-17T11:17:44.993641-0700 | INFO | mth5.groups.base | _add_group | RunGroup cut_eline already exists, returning existing group.
2024-10-17T11:17:45.019260-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id cut_eline. Setting to ch.run_metadata.id to cut_eline
2024-10-17T11:17:45.190917-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id cut_eline. Setting to ch.run_metadata.id to cut_eline
2024-10-17T11:17:45.381372-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id cut_eline. Setting to ch.run_metadata.id to cut_eline
2024-10-17T11:17:45.579565-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id cut_eline. Setting to ch.run_metadata.id to cut_eline
2024-10-17T11:17:45.822057-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id cut_eline. Setting to ch.run_metadata.id to cut_eline
<Figure size 640x480 with 2 Axes>
2024-10-17T11:17:48.761827-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id flipped_hx_hz. Setting to ch.run_metadata.id to flipped_hx_hz
2024-10-17T11:17:48.946316-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id flipped_hx_hz. Setting to ch.run_metadata.id to flipped_hx_hz
2024-10-17T11:17:49.023121-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id flipped_hx_hz. Setting to ch.run_metadata.id to flipped_hx_hz
2024-10-17T11:17:49.248386-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id flipped_hx_hz. Setting to ch.run_metadata.id to flipped_hx_hz
2024-10-17T11:17:49.347173-0700 | WARNING | mth5.groups.run | from_runts | Channel run.id 001 !=  group run.id flipped_hx_hz. Setting to ch.run_metadata.id to flipped_hx_hz
<Figure size 640x480 with 2 Axes>
2024-10-17T11:17:51.151002-0700 | INFO | mth5.mth5 | close_mth5 | Flushing and closing C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\mth5\data\mth5\test12rr.h5

Process with Aurora

Now that we’ve added noise lets see what it does to the MT response.

ap = AuroraProcessing()
ap.local_station_id = "test1"
ap.remote_station_id = None
ap.local_mth5_path = mth5_path
ap.remote_mth5_path = mth5_path

Run Summary

We need to get just the runs that we want to process. Include the remote run.

run_summary = ap.get_run_summary()
run_summary.mini_summary
Loading...

Here pick the runs you want to process. Should pick one of the noisy runs and then the remote. The index is the first column.

noise_run_summary = run_summary.clone()
noise_run_summary.df = noise_run_summary.df.iloc[[3, -1]]
noise_run_summary.mini_summary
Loading...

Process Single Station

Now process the noise type picked and see how that noise affects the data and how the remote reference helps mitigate that noise.

ap.remote_station_id = None
kds = ap.create_kernel_dataset(
    run_summary=noise_run_summary,
    local_station_id="test1",
    remote_station_id=None,
    )
config = ap.create_config(kds)
mt_12s_noise = ap.process_single_sample_rate(1, config, kds)
mt_12s_noise.plot_mt_response(plot_num=2)
2024-10-17T11:20:16.582945-0700 | INFO | mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column fc, adding and setting dtype to <class 'pandas._libs.missing.NAType'>.
2024-10-17T11:20:16.585177-0700 | INFO | mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column remote, adding and setting dtype to <class 'bool'>.
2024-10-17T11:20:16.586187-0700 | INFO | mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column run_dataarray, adding and setting dtype to <class 'NoneType'>.
2024-10-17T11:20:16.587186-0700 | INFO | mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column stft, adding and setting dtype to <class 'NoneType'>.
2024-10-17T11:20:16.588222-0700 | INFO | mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column mth5_obj, adding and setting dtype to <class 'NoneType'>.
2024-10-17T11:20:16.596185-0700 | INFO | aurora.config.config_creator | determine_band_specification_style | Bands not defined; setting to EMTF BANDS_DEFAULT_FILE
2024-10-17T11:20:16.633489-0700 | INFO | aurora.pipelines.transfer_function_kernel | show_processing_summary | Processing Summary Dataframe:
2024-10-17T11:20:16.639527-0700 | INFO | aurora.pipelines.transfer_function_kernel | show_processing_summary | 
   duration  has_data  n_samples            run station          survey       run_hdf5_reference   station_hdf5_reference    fc  remote  stft mth5_obj dec_level  dec_factor  sample_rate  window_duration  num_samples_window  num_samples  num_stft_windows
0   39999.0      True      40000  flipped_hx_hz   test1  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         0         1.0     1.000000            128.0                 128      39999.0             623.0
1   39999.0      True      40000  flipped_hx_hz   test1  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         1         4.0     0.250000            512.0                 128       9999.0             155.0
2   39999.0      True      40000  flipped_hx_hz   test1  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         2         4.0     0.062500           2048.0                 128       2499.0              38.0
3   39999.0      True      40000  flipped_hx_hz   test1  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         3         4.0     0.015625           8192.0                 128        624.0               8.0
2024-10-17T11:20:16.639527-0700 | INFO | aurora.pipelines.transfer_function_kernel | validate_processing | No RR station specified, switching RME_RR to RME
2024-10-17T11:20:16.640563-0700 | INFO | aurora.pipelines.transfer_function_kernel | validate_processing | No RR station specified, switching RME_RR to RME
2024-10-17T11:20:16.641561-0700 | INFO | aurora.pipelines.transfer_function_kernel | validate_processing | No RR station specified, switching RME_RR to RME
2024-10-17T11:20:16.641561-0700 | INFO | aurora.pipelines.transfer_function_kernel | validate_processing | No RR station specified, switching RME_RR to RME
2024-10-17T11:20:16.646584-0700 | INFO | aurora.pipelines.transfer_function_kernel | memory_check | Total memory: 31.83 GB
2024-10-17T11:20:16.647560-0700 | INFO | aurora.pipelines.transfer_function_kernel | memory_check | Total Bytes of Raw Data: 0.000 GB
2024-10-17T11:20:16.647560-0700 | INFO | aurora.pipelines.transfer_function_kernel | memory_check | Raw Data will use: 0.001 % of memory
2024-10-17T11:20:16.670776-0700 | INFO | aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: EMTF Synthetic, station: test1, run: flipped_hx_hz-- Fourier coefficients will be computed
2024-10-17T11:20:16.671979-0700 | INFO | mth5.mth5 | close_mth5 | Flushing and closing C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\mth5\data\mth5\test12rr.h5
2024-10-17T11:20:16.672217-0700 | INFO | aurora.pipelines.transfer_function_kernel | check_if_fcs_already_exist | FC levels not present
2024-10-17T11:20:16.672217-0700 | INFO | aurora.pipelines.process_mth5 | process_mth5_legacy | Processing config indicates 4 decimation levels
2024-10-17T11:20:16.672217-0700 | INFO | aurora.pipelines.transfer_function_kernel | valid_decimations | After validation there are 4 valid decimation levels
2024-10-17T11:20:17.184776-0700 | INFO | mtpy.processing.kernel_dataset | initialize_dataframe_for_processing | Dataset dataframe initialized successfully
2024-10-17T11:20:17.184776-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 0 Successfully
2024-10-17T11:20:17.408208-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hx with empty filters list detected
2024-10-17T11:20:17.409208-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:17.410208-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:17.455488-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hz with empty filters list detected
2024-10-17T11:20:17.455737-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:17.455737-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:17.455737-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:17.472818-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 25.728968s  (0.038867Hz)
2024-10-17T11:20:17.555029-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 19.929573s  (0.050177Hz)
2024-10-17T11:20:17.666323-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 15.164131s  (0.065945Hz)
2024-10-17T11:20:17.755530-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 11.746086s  (0.085135Hz)
2024-10-17T11:20:17.851958-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 9.195791s  (0.108745Hz)
2024-10-17T11:20:17.987649-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 7.362526s  (0.135823Hz)
2024-10-17T11:20:18.120246-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 5.856115s  (0.170762Hz)
2024-10-17T11:20:18.234581-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 4.682492s  (0.213562Hz)
2024-10-17T11:20:18.839270-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 1
2024-10-17T11:20:18.855640-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 1 Successfully
2024-10-17T11:20:19.057604-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hx with empty filters list detected
2024-10-17T11:20:19.070418-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:19.071430-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:19.110656-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hz with empty filters list detected
2024-10-17T11:20:19.110656-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:19.110656-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:19.110656-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:19.131693-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 102.915872s  (0.009717Hz)
2024-10-17T11:20:19.238173-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 85.631182s  (0.011678Hz)
2024-10-17T11:20:19.338771-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 68.881694s  (0.014518Hz)
2024-10-17T11:20:19.420025-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 54.195827s  (0.018452Hz)
2024-10-17T11:20:19.521539-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 43.003958s  (0.023254Hz)
2024-10-17T11:20:19.625901-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 33.310722s  (0.030020Hz)
2024-10-17T11:20:20.071988-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 2
2024-10-17T11:20:20.071988-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 2 Successfully
2024-10-17T11:20:20.268560-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hx with empty filters list detected
2024-10-17T11:20:20.271617-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:20.272491-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:20.304877-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hz with empty filters list detected
2024-10-17T11:20:20.305898-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:20.306884-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:20.307884-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:20.318886-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 411.663489s  (0.002429Hz)
2024-10-17T11:20:20.403537-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 342.524727s  (0.002919Hz)
2024-10-17T11:20:20.506877-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 275.526776s  (0.003629Hz)
2024-10-17T11:20:20.606282-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 216.783308s  (0.004613Hz)
2024-10-17T11:20:20.687756-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 172.015831s  (0.005813Hz)
2024-10-17T11:20:20.787933-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 133.242890s  (0.007505Hz)
2024-10-17T11:20:21.218678-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 3
2024-10-17T11:20:21.239078-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 3 Successfully
2024-10-17T11:20:21.405218-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hx with empty filters list detected
2024-10-17T11:20:21.405218-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:21.405218-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:21.442924-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hz with empty filters list detected
2024-10-17T11:20:21.442924-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:21.442924-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:21.442924-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:21.458766-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 1514.701336s  (0.000660Hz)
2024-10-17T11:20:21.556740-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 1042.488956s  (0.000959Hz)
2024-10-17T11:20:21.640011-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 723.371271s  (0.001382Hz)
2024-10-17T11:20:21.757987-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 532.971560s  (0.001876Hz)
2024-10-17T11:20:21.859571-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 412.837995s  (0.002422Hz)
2024-10-17T11:20:22.509148-0700 | INFO | mth5.mth5 | close_mth5 | Flushing and closing C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\mth5\data\mth5\test12rr.h5
<Figure size 960x720 with 5 Axes>
Plotting PlotMTResponse

Remote Reference Processing

Now we can try to reduce the effect of local noise by using a remote reference. In this case the remote helps only slightly because the time series is short and there aren’t that many estimates. If we had longer time series the remote would help more.

There are other things you can do to the data. You could apply a notch filter at the frequency your noise is at if you are certain that it is a pure sinusoid.

kds = ap.create_kernel_dataset(
    run_summary=noise_run_summary, 
    local_station_id="test1",
    remote_station_id="test2"
    )
config = ap.create_config(kds)
mt_12s_noise_rr = ap.process_single_sample_rate(1, config, kds)
mt_12s_noise_rr.plot_mt_response(plot_num=2)
2024-10-17T11:20:38.962685-0700 | INFO | mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column fc, adding and setting dtype to <class 'pandas._libs.missing.NAType'>.
2024-10-17T11:20:38.963652-0700 | INFO | mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column remote, adding and setting dtype to <class 'bool'>.
2024-10-17T11:20:38.964655-0700 | INFO | mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column run_dataarray, adding and setting dtype to <class 'NoneType'>.
2024-10-17T11:20:38.965653-0700 | INFO | mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column stft, adding and setting dtype to <class 'NoneType'>.
2024-10-17T11:20:38.966653-0700 | INFO | mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column mth5_obj, adding and setting dtype to <class 'NoneType'>.
2024-10-17T11:20:38.970716-0700 | INFO | aurora.config.config_creator | determine_band_specification_style | Bands not defined; setting to EMTF BANDS_DEFAULT_FILE
2024-10-17T11:20:39.023106-0700 | INFO | aurora.pipelines.transfer_function_kernel | show_processing_summary | Processing Summary Dataframe:
2024-10-17T11:20:39.023106-0700 | INFO | aurora.pipelines.transfer_function_kernel | show_processing_summary | 
   duration  has_data  n_samples            run station          survey       run_hdf5_reference   station_hdf5_reference    fc  remote  stft mth5_obj dec_level  dec_factor  sample_rate  window_duration  num_samples_window  num_samples  num_stft_windows
0   39999.0      True      40000  flipped_hx_hz   test1  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         0         1.0     1.000000            128.0                 128      39999.0             623.0
1   39999.0      True      40000  flipped_hx_hz   test1  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         1         4.0     0.250000            512.0                 128       9999.0             155.0
2   39999.0      True      40000  flipped_hx_hz   test1  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         2         4.0     0.062500           2048.0                 128       2499.0              38.0
3   39999.0      True      40000  flipped_hx_hz   test1  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>   False  None     None         3         4.0     0.015625           8192.0                 128        624.0               8.0
4   39999.0      True      40000            001   test2  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         0         1.0     1.000000            128.0                 128      39999.0             623.0
5   39999.0      True      40000            001   test2  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         1         4.0     0.250000            512.0                 128       9999.0             155.0
6   39999.0      True      40000            001   test2  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         2         4.0     0.062500           2048.0                 128       2499.0              38.0
7   39999.0      True      40000            001   test2  EMTF Synthetic  <HDF5 object reference>  <HDF5 object reference>  <NA>    True  None     None         3         4.0     0.015625           8192.0                 128        624.0               8.0
2024-10-17T11:20:39.023106-0700 | INFO | aurora.pipelines.transfer_function_kernel | memory_check | Total memory: 31.83 GB
2024-10-17T11:20:39.037030-0700 | INFO | aurora.pipelines.transfer_function_kernel | memory_check | Total Bytes of Raw Data: 0.001 GB
2024-10-17T11:20:39.038053-0700 | INFO | aurora.pipelines.transfer_function_kernel | memory_check | Raw Data will use: 0.002 % of memory
2024-10-17T11:20:39.053705-0700 | INFO | aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: EMTF Synthetic, station: test1, run: flipped_hx_hz-- Fourier coefficients will be computed
2024-10-17T11:20:39.054712-0700 | INFO | mth5.mth5 | close_mth5 | Flushing and closing C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\mth5\data\mth5\test12rr.h5
2024-10-17T11:20:39.057495-0700 | INFO | aurora.pipelines.transfer_function_kernel | mth5_has_fcs | Fourier coefficients not detected for survey: EMTF Synthetic, station: test2, run: 001-- Fourier coefficients will be computed
2024-10-17T11:20:39.057495-0700 | INFO | mth5.mth5 | close_mth5 | Flushing and closing C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\mth5\data\mth5\test12rr.h5
2024-10-17T11:20:39.070349-0700 | INFO | aurora.pipelines.transfer_function_kernel | check_if_fcs_already_exist | FC levels not present
2024-10-17T11:20:39.075688-0700 | INFO | aurora.pipelines.process_mth5 | process_mth5_legacy | Processing config indicates 4 decimation levels
2024-10-17T11:20:39.075688-0700 | INFO | aurora.pipelines.transfer_function_kernel | valid_decimations | After validation there are 4 valid decimation levels
2024-10-17T11:20:40.038855-0700 | INFO | mtpy.processing.kernel_dataset | initialize_dataframe_for_processing | Dataset dataframe initialized successfully
2024-10-17T11:20:40.038855-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 0 Successfully
2024-10-17T11:20:40.274243-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hx with empty filters list detected
2024-10-17T11:20:40.274243-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:40.274243-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:40.308143-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hz with empty filters list detected
2024-10-17T11:20:40.308143-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:40.308143-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:40.308143-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:40.621991-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:40.639222-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 25.728968s  (0.038867Hz)
2024-10-17T11:20:40.772274-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 19.929573s  (0.050177Hz)
2024-10-17T11:20:40.905704-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 15.164131s  (0.065945Hz)
2024-10-17T11:20:41.038512-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 11.746086s  (0.085135Hz)
2024-10-17T11:20:41.172541-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 9.195791s  (0.108745Hz)
2024-10-17T11:20:41.322588-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 7.362526s  (0.135823Hz)
2024-10-17T11:20:41.490110-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 5.856115s  (0.170762Hz)
2024-10-17T11:20:41.625019-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 4.682492s  (0.213562Hz)
2024-10-17T11:20:42.256885-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 1
2024-10-17T11:20:42.292742-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 1 Successfully
2024-10-17T11:20:42.490119-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hx with empty filters list detected
2024-10-17T11:20:42.490119-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:42.490119-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:42.528643-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hz with empty filters list detected
2024-10-17T11:20:42.528643-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:42.528643-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:42.528643-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:42.808201-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:42.835400-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 102.915872s  (0.009717Hz)
2024-10-17T11:20:42.957348-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 85.631182s  (0.011678Hz)
2024-10-17T11:20:43.092478-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 68.881694s  (0.014518Hz)
2024-10-17T11:20:43.207939-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 54.195827s  (0.018452Hz)
2024-10-17T11:20:43.342772-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 43.003958s  (0.023254Hz)
2024-10-17T11:20:43.474456-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 33.310722s  (0.030020Hz)
2024-10-17T11:20:43.909167-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 2
2024-10-17T11:20:43.947255-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 2 Successfully
2024-10-17T11:20:44.092047-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hx with empty filters list detected
2024-10-17T11:20:44.107717-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:44.108988-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:44.132211-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hz with empty filters list detected
2024-10-17T11:20:44.141280-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:44.142294-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:44.143319-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:44.403935-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:44.415161-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 411.663489s  (0.002429Hz)
2024-10-17T11:20:44.525366-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 342.524727s  (0.002919Hz)
2024-10-17T11:20:44.659142-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 275.526776s  (0.003629Hz)
2024-10-17T11:20:44.791602-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 216.783308s  (0.004613Hz)
2024-10-17T11:20:44.908999-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 172.015831s  (0.005813Hz)
2024-10-17T11:20:45.042381-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 133.242890s  (0.007505Hz)
2024-10-17T11:20:45.495179-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 3
2024-10-17T11:20:45.510260-0700 | INFO | aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 3 Successfully
2024-10-17T11:20:45.675202-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hx with empty filters list detected
2024-10-17T11:20:45.675829-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:45.676852-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:45.698764-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | Channel hz with empty filters list detected
2024-10-17T11:20:45.698764-0700 | WARNING | aurora.pipelines.time_series_helpers | calibrate_stft_obj | No filters to remove
2024-10-17T11:20:45.698764-0700 | WARNING | mt_metadata.timeseries.filters.channel_response | complex_response | No filters associated with <class 'mt_metadata.timeseries.filters.channel_response.ChannelResponse'>, returning 1
2024-10-17T11:20:45.709141-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:45.959866-0700 | INFO | aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
2024-10-17T11:20:45.976545-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 1514.701336s  (0.000660Hz)
2024-10-17T11:20:46.095601-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 1042.488956s  (0.000959Hz)
2024-10-17T11:20:46.226717-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 723.371271s  (0.001382Hz)
2024-10-17T11:20:46.351364-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 532.971560s  (0.001876Hz)
2024-10-17T11:20:46.476931-0700 | INFO | aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 412.837995s  (0.002422Hz)
2024-10-17T11:20:47.127444-0700 | INFO | mth5.mth5 | close_mth5 | Flushing and closing C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\mth5\data\mth5\test12rr.h5
2024-10-17T11:20:47.127444-0700 | INFO | mth5.mth5 | close_mth5 | Flushing and closing C:\Users\jpeacock\OneDrive - DOI\Documents\GitHub\mth5\mth5\data\mth5\test12rr.h5
<Figure size 960x720 with 5 Axes>
Plotting PlotMTResponse

DC Electric Train Noise

Depending on the active system, these can occupy a variety of frequency bands. Even when more than 100km away, these large amplitude fields can still affect MT transfer function estimates. Below a Figure from Kappler, 2010 that shows the decomposition of multivariate spectrograms into dominant eigenvalues as a function of frequency. The 1st and 2nd modes are associated with natural fields signal, and the later modes with other sources. Note the enhanced energy in the 3rd eigenmode at periods in the 20-40s bands. The signal is attributed to the Bay Area Rapid Transit (BART) train network.

from IPython.display import Image
Image("figures/bart_eigenvectors_time_series.png", width=1300)
<IPython.core.display.Image object>

This can be identified as cultural noise not only by eigenvalue amplitude and frequency band, but also because it has a weekly period, viz the reduction in signal amplitude on Sundays when the BART operates a reduced number of trains.

Image("figures/bart_eigenvectors_sundays.png", width=1300)
<IPython.core.display.Image object>

Pipeline noise

Pipelines often carry electric currents to reduce corrosion effects. These appear as narrow band signals. FOr example, below is a plot of the canonical coherence amplitudes for a small array of two MT station, showing the modes common between E and H fields. Note the elevated amplitude in the 3rd mode at around 4s period.

Image("figures/canonical_coherneces_EH_pkd_sao.png", width=1300)
<IPython.core.display.Image object>
aurora
Accessing Magnetic Field Data from EarthScope Using MTH5
aurora
Review of natural field transfer function algebra