marine-tracker / docs /clean.html
narniagooooo's picture
Initial commit: marine mammal tracking dashboard and route prediction
27b6a20
Raw History Blame Contribute Delete
6.43 kB
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Python: module clean</title>
</head><body>
<table class="heading">
<tr class="heading-text decor">
<td class="title">&nbsp;<br><strong class="title">clean</strong></td>
<td class="extra"><a href=".">index</a><br><a href="file:/Users/samuelveliveli/Downloads/marine-tracker/src/clean.py">/Users/samuelveliveli/Downloads/marine-tracker/src/clean.py</a></td></tr></table>
<p><span class="code">Clean&nbsp;noisy&nbsp;satellite&nbsp;tag&nbsp;fixes:&nbsp;speed&nbsp;filtering,&nbsp;deduplication,&nbsp;optional&nbsp;smoothing.<br>
&nbsp;<br>
Raw&nbsp;Argos/GPS&nbsp;fixes&nbsp;on&nbsp;marine&nbsp;mammals&nbsp;contain&nbsp;occasional&nbsp;large&nbsp;location&nbsp;errors<br>
that&nbsp;imply&nbsp;impossible&nbsp;travel&nbsp;speeds.&nbsp;This&nbsp;module&nbsp;applies&nbsp;the&nbsp;classic<br>
McConnell&nbsp;et&nbsp;al.&nbsp;(1992)&nbsp;speed-filtering&nbsp;approach&nbsp;used&nbsp;widely&nbsp;in&nbsp;marine<br>
telemetry:&nbsp;a&nbsp;forward&nbsp;pass&nbsp;per&nbsp;individual&nbsp;that&nbsp;drops&nbsp;any&nbsp;fix&nbsp;which&nbsp;would<br>
require&nbsp;traveling&nbsp;faster&nbsp;than&nbsp;a&nbsp;per-species&nbsp;plausible&nbsp;maximum&nbsp;from&nbsp;the&nbsp;last<br>
retained&nbsp;fix.</span></p>
<p>
<table class="section">
<tr class="decor pkg-content-decor heading-text">
<td class="section-title" colspan=3>&nbsp;<br><strong class="bigsection">Modules</strong></td></tr>
<tr><td class="decor pkg-content-decor"><span class="code">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span></td><td>&nbsp;</td>
<td class="singlecolumn"><table><tr><td class="multicolumn"><a href="numpy.html">numpy</a><br>
</td><td class="multicolumn"><a href="pandas.html">pandas</a><br>
</td><td class="multicolumn"></td><td class="multicolumn"></td></tr></table></td></tr></table><p>
<table class="section">
<tr class="decor functions-decor heading-text">
<td class="section-title" colspan=3>&nbsp;<br><strong class="bigsection">Functions</strong></td></tr>
<tr><td class="decor functions-decor"><span class="code">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span></td><td>&nbsp;</td>
<td class="singlecolumn"><dl><dt><a name="-clean_track"><strong>clean_track</strong></a>(df, max_speed_kmh=None, smooth=False)</dt><dd><span class="code">Full&nbsp;cleaning&nbsp;pipeline:&nbsp;dedupe&nbsp;-&gt;&nbsp;speed&nbsp;filter&nbsp;-&gt;&nbsp;optional&nbsp;Kalman&nbsp;smoothing.</span></dd></dl>
<dl><dt><a name="-haversine_km"><strong>haversine_km</strong></a>(lat1, lon1, lat2, lon2)</dt><dd><span class="code">Great-circle&nbsp;distance&nbsp;in&nbsp;km&nbsp;between&nbsp;two&nbsp;(lat,&nbsp;lon)&nbsp;points&nbsp;in&nbsp;decimal&nbsp;degrees.</span></dd></dl>
<dl><dt><a name="-kalman_smooth"><strong>kalman_smooth</strong></a>(df, process_noise=0.0001, measurement_noise=0.001)</dt><dd><span class="code">Apply&nbsp;a&nbsp;simple&nbsp;constant-velocity&nbsp;Kalman&nbsp;smoother&nbsp;to&nbsp;each&nbsp;individual's&nbsp;lat/lon&nbsp;track.<br>
&nbsp;<br>
Returns&nbsp;a&nbsp;copy&nbsp;of&nbsp;df&nbsp;with&nbsp;location_lat/location_long&nbsp;replaced&nbsp;by&nbsp;smoothed<br>
estimates.&nbsp;This&nbsp;is&nbsp;an&nbsp;optional&nbsp;pass&nbsp;--&nbsp;toggle&nbsp;raw&nbsp;vs.&nbsp;smoothed&nbsp;in&nbsp;the&nbsp;app<br>
rather&nbsp;than&nbsp;always&nbsp;applying&nbsp;it,&nbsp;since&nbsp;smoothing&nbsp;can&nbsp;mask&nbsp;genuine&nbsp;sharp<br>
turns.</span></dd></dl>
<dl><dt><a name="-remove_duplicates"><strong>remove_duplicates</strong></a>(df, min_time_diff_seconds=60, min_dist_m=1.0)</dt><dd><span class="code">Drop&nbsp;near-duplicate&nbsp;consecutive&nbsp;fixes&nbsp;for&nbsp;the&nbsp;same&nbsp;individual.<br>
&nbsp;<br>
A&nbsp;fix&nbsp;is&nbsp;considered&nbsp;a&nbsp;near-duplicate&nbsp;of&nbsp;the&nbsp;previous&nbsp;one&nbsp;if&nbsp;it&nbsp;arrived<br>
within&nbsp;min_time_diff_seconds&nbsp;AND&nbsp;within&nbsp;min_dist_m&nbsp;--&nbsp;i.e.&nbsp;it&nbsp;adds&nbsp;no&nbsp;new<br>
information&nbsp;about&nbsp;position&nbsp;or&nbsp;timing.&nbsp;Iterates&nbsp;over&nbsp;numpy&nbsp;arrays&nbsp;rather<br>
than&nbsp;DataFrame&nbsp;rows&nbsp;since&nbsp;this&nbsp;forward-dependent&nbsp;pass&nbsp;(each&nbsp;decision<br>
depends&nbsp;on&nbsp;the&nbsp;last&nbsp;*retained*&nbsp;fix)&nbsp;can't&nbsp;be&nbsp;fully&nbsp;vectorized,&nbsp;and<br>
per-row&nbsp;Series&nbsp;construction&nbsp;is&nbsp;the&nbsp;dominant&nbsp;cost&nbsp;on&nbsp;studies&nbsp;with&nbsp;100k+&nbsp;fixes.</span></dd></dl>
<dl><dt><a name="-speed_filter"><strong>speed_filter</strong></a>(df, max_speed_kmh=None)</dt><dd><span class="code">Drop&nbsp;fixes&nbsp;that&nbsp;imply&nbsp;travel&nbsp;speed&nbsp;above&nbsp;a&nbsp;per-species&nbsp;plausible&nbsp;maximum.<br>
&nbsp;<br>
Forward&nbsp;pass&nbsp;per&nbsp;individual&nbsp;(McConnell&nbsp;et&nbsp;al.&nbsp;speed-filtering&nbsp;approach):<br>
walk&nbsp;fixes&nbsp;in&nbsp;time&nbsp;order,&nbsp;keeping&nbsp;a&nbsp;fix&nbsp;only&nbsp;if&nbsp;its&nbsp;implied&nbsp;speed&nbsp;from&nbsp;the<br>
last&nbsp;*retained*&nbsp;fix&nbsp;is&nbsp;at&nbsp;or&nbsp;below&nbsp;the&nbsp;threshold.&nbsp;max_speed_kmh&nbsp;may&nbsp;be&nbsp;a<br>
single&nbsp;float&nbsp;(applied&nbsp;to&nbsp;all&nbsp;individuals)&nbsp;or&nbsp;None&nbsp;to&nbsp;look&nbsp;up&nbsp;each&nbsp;row's<br>
species&nbsp;in&nbsp;SPECIES_MAX_SPEED_KMH&nbsp;(falling&nbsp;back&nbsp;to&nbsp;DEFAULT_MAX_SPEED_KMH).<br>
Iterates&nbsp;over&nbsp;numpy&nbsp;arrays&nbsp;rather&nbsp;than&nbsp;DataFrame&nbsp;rows&nbsp;for&nbsp;the&nbsp;same&nbsp;reason<br>
as&nbsp;remove_duplicates:&nbsp;the&nbsp;greedy&nbsp;dependency&nbsp;prevents&nbsp;full&nbsp;vectorization,<br>
but&nbsp;avoiding&nbsp;per-row&nbsp;Series&nbsp;construction&nbsp;is&nbsp;still&nbsp;much&nbsp;faster.</span></dd></dl>
</td></tr></table><p>
<table class="section">
<tr class="decor data-decor heading-text">
<td class="section-title" colspan=3>&nbsp;<br><strong class="bigsection">Data</strong></td></tr>
<tr><td class="decor data-decor"><span class="code">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span></td><td>&nbsp;</td>
<td class="singlecolumn"><strong>DEFAULT_MAX_SPEED_KMH</strong> = 15.0<br>
<strong>EARTH_RADIUS_KM</strong> = 6371.0088<br>
<strong>SPECIES_MAX_SPEED_KMH</strong> = {'Balaena mysticetus': 15.0, 'Balaenoptera musculus': 20.0, 'Balaenoptera physalus': 20.0, 'Halichoerus grypus': 20.0, 'Megaptera novaeangliae': 15.0, 'Phoca vitulina': 20.0, 'Trichechus manatus': 10.0}</td></tr></table>
</body></html>