{
"cells": [
{
"cell_type": "raw",
"metadata": {
"raw_mimetype": "text/restructuredtext"
},
"source": [
".. index::\n",
" single: InfluxDB; Streamer\n",
".. index::\n",
" single: Instruments; InfluxDB Out"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# InfluxDB Out"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"_DashT InfluxDB Out_ is an \"instrument\" which is used to start and monitor a background streamer which takes all or selected data received or created by _DashT_ and formats that data so that it is compatible with [InfluxDB v2.0 time series database](../influxdb/influxdb.ipynb#InfluxDB-/-Grafana)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"InfluxDB [Line Protocol to write data](https://v2.docs.influxdata.com/v2.0/write-data/) is used. Users can select from two different methods: either direct streaming into InfluxDB 2.0 HTTP server or writing to a Line Protocol-format file which is can be later imported in the database using InfluxDB 2.0 web user interface."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since the Line Protocol-format file method does not require the presence of [InfluxDB v2.0](../influxdb/influxdb.ipynb#InfluxDB-/-Grafana) server instance while underway, it is easier to understand so let's explain it first."
]
},
{
"cell_type": "raw",
"metadata": {
"raw_mimetype": "text/restructuredtext"
},
"source": [
".. index::\n",
" single: InfluxDB; Data File"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Line Protocol File"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is the easiest way to get started since no running Influx DB 2.0 server is needed while data is collected. It is also the fastest way to collect data for InfluxDB, limited only by the writing speed of your hard disk. Potentially, but depending of your boat's instruments, over thirty different types of measurement values received by various Dashboard Intstruments can be saved, some of them a few times per second. Since modern disks can easily store data with these rates and volumes, it is the safest option what comes to the performance and the ease of use."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Start your OpenCPN v5.2 or superior with _DashT_ plug-in."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Open the _DashT_ Preferences dialog and create a **new instrument panel with a single instrument in it**: _InfluxDB Out_:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"img/s_005_DashT_idbout_create_pane_and_instru.png\"\n",)
[(zoom)](img/005_DashT_idbout_create_pane_and_instru.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
">**NOTE**: Albeit you can have many _InfluxDB Out_ instruments, only the first one is actually doing any streaming - the others would only do monitoring. It can be confusing to remember which one is doing the actual work and one can accidentally delete it... Therefore it is advised to have only one _InfluxDB Out_ instrument, alone in a dedicated instrument panel: if you mix _InfluxDB Out_ with other instruments, and then reorganize those other instruments, the communication and streaming thread can get orphan below the _InfluxDB Out_ while the panel gets reorganized and would halt. There is no way out of the halt condition other than restarting _OpenCPN_."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"img/s_010_DashT_idbout_created.png\"\n",)
[(zoom)](img/010_DashT_idbout_created.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That's it! Data coming in from OpenCPN (there is always some, like it is giving position data being ) is already collected and a background thread is busy writing those into a file."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"img/s_015_DashT_idbout_steaming_to_file.png\"\n",)
[(zoom)](img/015_DashT_idbout_steaming_to_file.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looks small? Well, that's hundreds of lines written out, after a minute or so 300 lines have been written..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### File content"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Below is an example of InfluxDB Line Protocol file (small part of it only after a few seconds of running) created by _InfluxDB Out_. It can be dropped directly into the [InfluxDB v2.0 data collection user interface](https://v2.docs.influxdata.com/v2.0/write-data/#add-a-demo-data-bucket) as it is."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
">**NOTE**: _InfluxDB v2.0_ will asks you the precision of the timestamps: it is in **milliseconds** as you can see in the data of the excerpt below."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"...\n",
"navigation magneticVariation=-4.000000 1598810821136\n",
"navigation headingMagnetic=145.000000 1598810821136\n",
"navigation headingTrue=141.000000 1598810821136\n",
"navigation speedThroughWater=0.000000 1598810821140\n",
"environment,prop1=temperature water=21.600000 1598810821140\n",
"environment,prop1=belowTransducer depth=15.400000 1598810821384\n",
"environment,prop1=directionTrue wind=20.000000 1598810821384\n",
"environment,prop1=speedTrueGround,prop2=tws wind=12.200000 1598810821384\n",
"...\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You may be wondering about the format, where there is a resemblance with Signal K data. That is true, Signal K data model is used but what you see is created by a _DashT_ _InfluxDB v2.0_ [database schema]( https://en.wikipedia.org/wiki/Database_schema). The purpose of this schema is to make both writing and reading of data more efficient, one would be drilling first to `environment` data, then `wind` and only after seeking `directionTrue`. This allows data to be organized in continuous chunks, thus speeding the operation and reducing the number of reads needed to reach the value searched for."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
">**NOTE**: If you are a database specialist or otherwise do not agree with the _DashT_ _InfluxDB v2.0_ database schema, the good news is that it is totally configurable in the JSON-format configuration file described below. One can, if it is the wish, to move use single Signal K key in CamelCase, in order to be compatible with _Signal K server node_ way of writing data in a database. No tests, however, have been carried out without applying a schema in write / read operations."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### File location"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Windows**: `C:\\ProgramData\\opencpn\\plugins\\dashboard_tactics_pi\\streamout`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Linux**: `~/.opencpnplugins/dashboard_tactics_pi/streamout`"
]
},
{
"cell_type": "raw",
"metadata": {
"raw_mimetype": "text/restructuredtext"
},
"source": [
".. index::\n",
" single: InfluxDB; Data backups"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Backup copies"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"At each start of _InfluxDB Out_ an existing `streamout` InfluxDB 2.0 line protocol file is renamed as `streamout_bup_YYYY-MM-DD_hh:mm:ss` where the `YYYY-MM-DD_hh:mm:ss` is the time of the backup and **not** the time of the data inside that file:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To find out the period of the data **inside** a backup file open it an use on-line service accepting timestamps in milliseconds such as https://www.epochconverter.com to convert the timestamp of the first and last line. "
]
},
{
"cell_type": "raw",
"metadata": {
"raw_mimetype": "text/restructuredtext"
},
"source": [
".. index::\n",
" single: InfluxDB; Line data import"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Importing to InfluxDB"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"InfluxDB has multiple file import sources available in its graphical user interface. Typically you would select `Load Data -> Buckets -> Add data` and then drag and drop the line data file into the file selection box."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
">**NOTE**: The above would work without a problem for a file registered during a regatta or such. Files registered over longer period may grow too big for the InfluxDB network based file loader (limit is, as this is written **10MB** file maximum). In this case one need to split the big file smaller. For Windows, one can use simple [FileSplitter](https://sourceforge.net/projects/fsplit/files/latest/download), while for Linux on can say `split -l 80000 input_file output_file`. To give you and idea, 48 hours of continuous navigation creates a line protocol file of size 50MB, approximately."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
">**CAVEAT**: in splitting a line protocol file by megabytes (size) the last line of the each file and first line of the next one are split blindly in two and, consequently InfluxDB would detect a line protocol error and refuse to upload the file. The solution is to cut the last, incomplete line of each file and complete the first line of the next one with it. This sounds cumbersome but it is not that much of work. However, prefer the per line splitting method if you can."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Configuration file management"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The configuration file and the database schema are in a single JSON-file, located in the same data directory indicated above. It can be edited. There is a template file located in the plug-in's program directory dashboard_tactics_pi from where it should be copied (and not modified) if a need arise - see below for HTTP template file for more details."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is possible to have more than one configuration settings and, consequently more than one database schema - you can add tags, or otherwise modify the schema to facilitate the data browsing. You can also switch between file-based and http:// based streaming operation. But only one configuration file can be used at a time. Select its name in the [ini/conf file](../tweaks/tweaks.ipynb#Streamout)."
]
},
{
"cell_type": "raw",
"metadata": {
"raw_mimetype": "text/restructuredtext"
},
"source": [
".. index::\n",
" single: InfluxDB; HTTP streamout"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## HTTP Streamout"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"HTTP Streamout into InfluxDB v2.0 is needed if you are using Grafana visualization when underway, or if you want to use _DashT_ [Line Charts](../linechart/linechart.ipynb#Line-Chart) which are Grafana type (modestly imitating) instruments but embedded in Dashboard, using data retrieved from InfluxDB v2.0 database and applying aggregation functions implemented by InfluxDB v2.0 on that data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Make a copy of a template file `streamout_template_http.json` from:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Windows**: `C:\\Program Files (x86)\\OpenCPN\\plugins\\dashboard_tactics_pi\\data`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Linux**: `./share/opencpn/plugins/dashboard_tactics_pi/data`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and place it here, with this name (or change the name to your liking):"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Windows**: `C:\\ProgramData\\opencpn\\plugins\\dashboard_tactics_pi\\streamout_http.json`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Linux**: `~/.opencpnplugins/dashboard_tactics_pi/streamout_http.json`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Select the above path and name in the [ini/conf file](../tweaks/tweaks.ipynb#Streamout)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Modify `streamout-http.json` file to contain your InfluxDB 2.0 connection parameters, see [Set Up InfluxDB](../influxdb/influxdb.ipynb#Set-up-InfluxDB):"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
" \"org\" : \"myorg\", // HTTP: Influx DB organization name\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You have seleced an organization (boat?) name in InfluxDB 2.0. Type it here in place of `myorg` - exactly as you defined it in InfluxDB 2.0 user interface."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
" \"bucket\" : \"mybucket\", // HTTP: Influx DB bucket to write\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Data is written in buckets of InfluxDB 2.0. Create one if not yet done and give its exact name here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
" \"token\" : \"ToLdk3DNs3PqbKNS2hdZMure......E0eu4lE0OUWRt8w==\"\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Tokens are unique, per each server, and sometimes even per each bucket in it. Go to Tokens-tab in Settings of InfluxDB 2.0 user interface and generate one. Copy the new token into clipboard and paste it between string quotes in the configuration file. Do not allow a carriage return, it must be a **single line string**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now you are ready to try. But if it does not work? There is debugging information available in OpenCPN log file, read more about debugging below. But before that you must turn the debugging on in the configuration file and restart the _DashT InfluxDB Out_ \"instrument\" (often it is easier to restart the OpenCPN):"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`\"verbosity\" : 3 // 0=quiet,1=events,2=verbose,3+=debug`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conversions"
]
},
{
"cell_type": "raw",
"metadata": {
"raw_mimetype": "text/restructuredtext"
},
"source": [
".. index::\n",
" single: InfluxDB; VDR, from"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Stream from VDR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is possible to convert old OpenCPN [Voyage Data Recorder plug-in](https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:opencpn_user_manual:plugins:logs:vdr) log files into a historically time stamped data in InfluxDB v2.0 time series database by making a VDR data play back."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following major criteria must be fulfilled:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* The data contains GPS timestamps\n",
"* You do not have [Calculate SOG- option in OpenCPN Preferences checked](https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:opencpn_user_manual:plugins:logs:vdr#faq)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The resulting database entries are organized with timestamps generated by _DashT_ which gets its time synchronized at every arrival of the GPS data with a time stamp. In between, the time tick is running with the CPU clock, until getting synchronized again. While the intermediate data arrival rate is dictated by the playback speed in the VDR player, the error in successive data sentences is not cumulative throughout the file, only between two GPS time data sentences."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
">**NOTE**: The time shift backwards is possible because if _DashT_ detects a difference greater than five seconds between the local CPU time and GNSS (like GPS) provided time it switches to use the GPS time for timestamps and calculations. This shift is announced in the OpenCPN log file only, there is no pop-up message - taking it as granted that in off-shore use the racer is not necessarily synchronizing his/her PC with the satellite provided time anyway. Depending of the ratio of the time data arrival, this may lead to a short range with time values being the current (CPU) time before the data is timestamped with the \"historical\" timestamps. If this is an issue, remove those few tens of lines from the line data file manually, before uploading into the InfluxDB v2.0. "
]
},
{
"cell_type": "raw",
"metadata": {
"raw_mimetype": "text/restructuredtext"
},
"source": [
".. index::\n",
" single: InfluxDB; Debugging"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Debugging"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The output file, _streamout_ is a text file which can be visualized and its filling can be followed up with the command line tools.\n",
"\n",
"It is possible to increase also the verbosity level of the log messages, values 3 - 5 makes more debug printing out in the OpenCPN log file."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Level 4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Verbosity level 4 is printing out every second the status of the FIFO queue between the streamer instrument and the file writing and data communication thread, which is useful if buffer overrun is suspected"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"On Windows PowerShell:\n",
"\n",
"```\n",
"PS C:\\ProgramData\\opencpn\\plugins\\dashoard_tactics_pi>\n",
"Get-Content ./streamout -Wait -Tail 20\n",
"```\n",
"```\n",
"PS C:\\ProgramData\\opencpn>\n",
"Get-Content ./opencpn.log -Wait -Tail 20\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"On *nix systems:\n",
"\n",
"```\n",
"tail -f ~.opencpnplugins/dashboard_tactics_pi/streamout\n",
"```\n",
"```\n",
"tail -f ~.opencpn/opencpn.log\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Level 5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Level 5 is useful to debug issues with the HTTP, printing out the entire header and data POST/OPTIONS message (one may want to **reduce the number of lines** to be sent per message in this case)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.2"
}
},
"nbformat": 4,
"nbformat_minor": 4
}