Skip to article frontmatterSkip to article content
from IPython.display import Image
# @hidden_cell
Image("figures/00_aurora_backdrop_VbZDFnn.jpeg")
<IPython.core.display.Image object>
Image("figures/00a_aurora_backdrop_screenshot_with_credits.png")
<IPython.core.display.Image object>

Overview

  • Objectives, Motivation, Core Packages, Development Principles, History & Software testing

Processing Outline

  • TF Calculations (external notebook)

  • High level flow, TF Kernel Concept, Main Interfaces

Processing Detail

  • Code Flows

  • Core data structures

Example Results

  • Synthetic Data

  • Field Data

Jupyter Notebook exercises

  • Synthetic Data processing

  • Earthscope Data

Here are some notes ...

Aurora

  • EarthScope (IRIS) Sponsored Project

  • Hosted by SimPEG (Geophysical tools developed at UBC)

Objectives:

  • Provide Open-Source Magnetotelluric (MT) Time Series Processing in Python

  • Produce transfer function (TF) objects in mt_metadata format from time series archived in mth5 format.

  • Emulate Egbert’s EMTF legacy remote-reference FORTRAN and Matlab codes

Motivation

Allow researchers to spend more time on data analysis and interpretation, and less time worrying about:

  • interfaces between metadata and processing

  • time series wrangling

  • digital filtering

  • spectral leakage

  • spectrogram transformations

  • calibration (digital counts to SI units)

  • complex-valued regression, regularization, etc.

  • robust statistics

  • noise level and error bar estimation

  • rotations

Provide framework for:

  • plugging-in new methods

  • comparison of benchmark results with updated

  • simple exploration of processing configurations

Image("figures/xkcd_standards.png")
<IPython.core.display.Image object>

Core Packages

  • numpy

  • pandas

  • scipy

  • xarray

Future versions planned to use dask and numba

Development Principles

  • Open source

  • Leverage existing tools and frameworks

  • Extensibility

  • Reusability

  • Readability (PNCA)

  • Parallelizability*

*Work in progress

History

  • 2021: Translate EMTF to python and work with MTH5

  • 2022: Workflows and examples, development

  • 2023: Wide scale testing

  • 2024: Fourier Coefficients layer tested, features layer in development

Image("figures/01_joss_manuscript_cover.png")
<IPython.core.display.Image object>

Software Testing

  • Relatively well tested

    • Uses a combination of unit-tests and integrated tests

Tests rely on Real and Synthetic Data:

  • EMTF came packaged with synthetic data time series from two stations, and we used these data to validate aurora’s results.

  • Provides a benchmark

  • Used for rapid integrated tests

Image("figures/github_tested.png")
<IPython.core.display.Image object>

MT Data Processing Outline

The Transfer function equations are described in terms of processing in a companion notebook:

auxiliary/tf_calculations.ipynb

(These links may work in jupyter-lab)

Image("figures/data_processing_ecosystem.png")
<IPython.core.display.Image object>

High Level Processing Flow

Image("../../../aurora/paper/aurora_workflow.png", width=3000, height=4400) # figures/00a_aurora_backdrop_screenshot_with_credits.png")
<IPython.core.display.Image object>
Image("figures/process_flow_high_level_paper_figure_2.png")
<IPython.core.display.Image object>
Image("figures/tf_kernel_paper_figure_1.png")
<IPython.core.display.Image object>
Image("figures/process_flow_main_interfaces.png")
<IPython.core.display.Image object>

Detailed Processing Flow

Image("figures/detailed_flow_1_of_2.png")
<IPython.core.display.Image object>
Image("figures/detailed_flow_2_of_2.png")
<IPython.core.display.Image object>

Image("figures/core_data_structures.png")
<IPython.core.display.Image object>
Image("figures/run_summary.png")
<IPython.core.display.Image object>
Image("figures/kernel_dataset.png")
<IPython.core.display.Image object>

Synthetic Data Processing Results

Image("figures/synthetic_rho_phi_curves.png")
<IPython.core.display.Image object>
Image("figures/synthetic_tabulated_results.png")
<IPython.core.display.Image object>
Image("figures/minimal_code_example.png")
<IPython.core.display.Image object>
Image("figures/tf_comparison_cas04_with_remote_reference.png")
<IPython.core.display.Image object>
Image("figures/widescale_tests_histograms.png")
<IPython.core.display.Image object>

Recent Updates

  • Processing Summary and Processing Pre-check:

    • Processing Summary “melts” decimation levels over runs in Kernel Dataset

    • One row per (decimation_level, run) pair

    • Validiaty checks on each row before processing to ensure consistency between Data and Processing Parameters

    • Provides an iterator for building FC layer

  • Fourier Coefficient (“FC”) Group in MTH5:

    • Removes need to re-compute if testing weights and regression schemes

    • Aurora checks for stored FCs consistent with the Processing Configuration

      • If present, they are used for TF estimation

  • Wide scale testing on >1000 Time Series and TFs archived at Earthscope

  • PyPI release: pip install aurora=0.4.2

  • Now installs as part of MTPy

Limitations

  • Can only use a one remote reference station Vs. multiple station simultaneous processing (full matrix of spectral cross power)

  • Assumes 3-,4-,or 5-channel stations

  • Limited data cleaning algorithms

    • Huber’s Regression M-Estimate implemented,

    • Time domain cleaning not implemented

  • Limited data visualization capability

  • No handling for gaps / interpolation

Future Work

  • Plug-ins for:

    • other open source MT codes, birrp-based TFs, etc.

    • Weights: coherency sorting, polarization sorting, etc.

    • ARMA prewhitening

    • Data Imputation

  • Multiple station processing regression engine

  • Expand testing suite:

    • Multiple benchmark datasets

    • More synthetic data cases

  • Align with IAGA Div VI TaskForce for Data interoperability

  • Community testing on benchmark datasets

  • HPC Applications

Longer term future work

  • Moving platform data

  • Allow Natural Fields processing of Induced polarization and DC resistivity layouts

Conclusion:

Aurora provides an example workflow for obtaining robust TF estimates from data archived in MTH5.

Get involved:

Back Up Slides

Image("figures/mt_sources_and_amplitudes.png")
<IPython.core.display.Image object>
Image("figures/standards_proliferate_translators.png")
<IPython.core.display.Image object>
Image("figures/what_do_the_time_series_look_like.png")
<IPython.core.display.Image object>
Image("figures/why_three_packages.png")
<IPython.core.display.Image object>
Image("figures/cascading_decimation.png")
<IPython.core.display.Image object>
Image("figures/potpourri.png")
<IPython.core.display.Image object>
Image("figures/schumann_01.png")
<IPython.core.display.Image object>
Image("figures/schumann_02.png")
<IPython.core.display.Image object>
Image("figures/why_three_packages.png")
<IPython.core.display.Image object>
Image("figures/recent_updates.png")
<IPython.core.display.Image object>
Image("figures/conclusion.png")
<IPython.core.display.Image object>
!ls figures/*png
figures/00a_aurora_backdrop_screenshot_with_credits.png
figures/01_joss_manuscript_cover.png
figures/aurora_backdrop.png
figures/cascading_decimation.png
figures/conclusion.png
figures/core_data_structures.png
figures/data_processing_ecosystem.png
figures/detailed_flow_1_of_2.png
figures/detailed_flow_2_of_2.png
figures/github_tested.png
figures/kernel_dataset.png
figures/minimal_code_example.png
figures/mt_sources_and_amplitudes.png
figures/potpourri.png
figures/process_flow_high_level_paper_figure_2.png
figures/process_flow_main_interfaces.png
figures/recent_updates.png
figures/run_summary.png
figures/schumann_01.png
figures/schumann_02.png
figures/standards_proliferate_translators.png
figures/synthetic_rho_phi_curves.png
figures/synthetic_tabulated_results.png
figures/tf_comparison_cas04_with_remote_reference.png
figures/tf_kernel_paper_figure_1.png
figures/what_do_the_time_series_look_like.png
figures/why_three_packages.png
figures/widescale_tests_histograms.png
figures/xkcd_standards.png
mth5
Example 10: Full workflow with MTpy-v2 as the Driver
aurora
Build an MTH5 and Operate the Aurora Pipeline