Learning about AIS data cleansing methods

How Spire Sense addresses errors, empty values & other issues in received AIS messages

General Data Cleansing Methods

Often times raw AIS messages can contain errors, empty or default values, and points over land due to GPS issues or spoofing.

For the feed served up via API, we do our best to clean out some of these errors. Below is a list of some of the steps that we take to clean the data:

  • Check values against the AIS standard.
  • Remove position data with unavailable coordinates (91 / 181).
  • Filter out messages with invalid MMSI numbers. (Ships should report 9-digit MMSIs. Base stations should report 7-digit MMSIs.)
  • Filter out messages with IMO numbers more than 7 digits. Note 7 digit IMO numbers still returned even when not matching the IMO checksum calculation.
  • Positions reported over land. (Note. feature enabled in data from 2019-08-27)

It is easy to get the full feed, including errors, by simply attaching the cleansed=false  parameter within calls to the Messages API.

Cleaned by default

Note that the Messages API implements the above data cleaning rules by default. To disable the cleaning user the filter cleansed=false when calling Messages API.

Currently we do not perform cleansing for:

  • Ships reporting an unintelligible name or destination field
  • Ships reporting inappropriate dimensions or speeds. 
  • Other data errors.

The TCP feed is the rawest form we provide, which does not include data cleansing.

Below is a plot of AIS reported positions that are excluded by the Spire Land Filter.