The data mining structure of nodes controls the bots you need to run to extract data from exchanges and process it into indicators or technical studies.
Data mining is the activity of processing data. You need to process data to feed charts, and so that your trading systems may make decisions based on quality information.

Having access to quality information is a crucial element of trading. For that reason, Superalgos strives to give users full control over how, where, and when data is processed. You will use this section of the network hierarchy to exert that control.

Data mining is done through tasks running instances of sensor and indicator bots. The data structure under this node allows organizing the data mining operation by exchange, market, and data mine.

The setup of the data mining operation may be done using the Install Market function of the market you wish to install as defined in the Crypto Ecosystem hierarchy under the exchange markets node. Learn more about this function on the how to install a new market page.

You may also set up data mining tasks by using the different functions in this section of the hierarchy, starting with the options in the menu of this node. This route offers finer control over what tasks you set up, and how.


Data Mining

Exchange Data Tasks

Market Data Tasks

Data Mine Tasks

Task Manager

Task

Sensor Bot Instance

Sensor Process Instance

Indicator Bot Instance

Indicator Process Instance

Time Frames Filter

Key Reference

Exchange Data Tasks

The exchange data tasks node organizes data mining tasks by exchange. That is, each exchange installed in the system has an exchange data tasks node grouping all market data tasks corresponding to the said exchange.

The exchange data tasks node must reference the exchange of choice. This reference constraints the rest of the definitions to the context of the said exchange.

When representing an exchange featured in the system’s icons library, the standard exchange data tasks icon is replaced by the exchange’s logo.

Click to learn more about exchange data tasks

Adding an Exchange Data Tasks Node

To add a specific exchange data tasks node, select Add Exchange Data Tasks on the parent node menu.

You may also add exchange data tasks node in bulk for all exchanges that may have been previously defined in the Crypto Ecosystem hierarchy. To do that, select Add Missing Exchanges on the parent node menu. An exchange data tasks node is created for each defined exchange, each with the corresponding reference.

Starting an Exchange Data Tasks Node

Select Run All Market Data Tasks or Stop All Market Data Tasks on the menu to start and stop all tasks under this node.

Market Data Tasks

A market data tasks node groups data mining tasks operating in a specific market.

The market data tasks node must reference a market defined in the Crypto Ecosystem hierarchy.

This node may spawn individual data products or may deploy data products in bulk organized by data mine and by bots. See the data products node for the details.

Click to learn more about market data tasks

Adding a Market Data Tasks Node

To add a market data tasks node, select Add Market Data on the parent node menu.

You may also add market data tasks nodes in bulk for all markets that may have been previously defined in the Crypto Ecosystem hierarchy. To do that, select Add Missing Markets on the parent node menu. A market data tasks node is created for each defined market, each with the corresponding reference.

Starting a Market Data Tasks Node

Select Run All Data Mine Tasks or Stop All Data Mine Tasks on the menu to start and stop all tasks under this node.

Data Mine Tasks

The data mine tasks node is an organizational device that groups tasks corresponding to the referenced data mine.

The data mine tasks node must reference the definition of a data mine. The node may spawn tasks for each bot in the data mine.

Click to learn more about data mine tasks

Adding a Data Mine Tasks Node

To add a data mine tasks node, select Add Data Mine Tasks on the parent node menu. This action adds the node but does not establish a reference with any data mine.

The smarter use of the node involves using the Add Missing Data Mine Tasks option on the parent node menu. This action creates a data mine tasks node for each data mine in the workspace, establishing a reference with the corresponding data mines. This is the first step in the direction of quickly setting up tasks for each bot in a given data mine.

Starting a Data Mine Tasks Node

Select Run All Task Managers or Stop All Task Managers on the menu to start and stop all tasks under this node.

Task Manager

A task manager is a device used to organize and control any number of tasks, which, in turn, control bot instances. You use a task manager to start or stop several tasks at the same time.

Task managers are used both in the context of data mining and trading operations, to facilitate the organization of tasks.

For example, you may set up a task manager to handle tasks related to a particular set of indicators used with a certain strategy. Or, for example, to organize multiple backtesting sessions.

Click to learn more about task managers

Adding a Task Manager Node

To add a task manager, select Add Task Manager on the parent node menu.

Starting a Task Manager

Select Run All Tasks or Stop All Tasks on the menu to start and stop all tasks respectively.

Task

A task is the device used to control bot instances, that is, to start and stop bots, including sensors, indicators and the trading bot.

Each task controls a single bot. A bot instance running on its own task is independent from other bots at the operating system level, thus, may not be affected by errors ocurring on those other bots.

Click to learn more about tasks

Adding a Task Node

To add a task, select Add Task on the task manager node menu.

Starting a Task

Select Run on the menu to start a task. When a task is started, the process instance of the bot instance attached to the task is started. Also, a visual indication that both the task and the process instance are running appear surrounding the corresponding nodes, in the form of a progress ring.

To stop a task, select Stop on the menu.

Sensor Bot Instance

A sensor bot instance is a reference to a sensor bot as defined in a data mine. The instance of the bot runs the defined processes and generates the defined data products.

A sensor bot is an algorithm that extracts raw data from external sources (i.e.: exchanges, Twitter, etc.) and stores it in a dataset that other bots may consume.

The sensor bot instance holds no definitions as to what the bot does. Instead, its process instance references a process definition in the corresponding data mine. That is how the sensor bot instance obtains the information regarding what it needs to do once it is run.

Click to learn more about sensor bot instances

Adding a Sensor Bot Instance Node

To add a sensor bot instance, select Add Sensor Bot Instance on the task node menu.

Configuring the Sensor Bot Instance

Select Configure Sensor Bot Instance on the menu to access the configuration.

  {
    "startDate": "2020-01-01"
  }
  • startDate is the desired starting date of the data product the sensor bot instance builds, in the YYYY-MM-DD format. The sensor bot instance queries its data source for data starting on the configured startDate.

    • The actual date in which the dataset starts depends on external factors: A. The market may start at a later date. B. The exchange may limit how far in the past data may be retrieved. In both cases, the sensor bot automatically discovers the date closest to the desired starting date that is possible to start with, and proceeds accordingly.

    • In the case the startDate is changed after the sensor bot has started building a data product, either for an earlier or later date, the sensor re-evaluates the feasibility of starting at the new date. The actual date may or may not change; regardless, the sensor bot discards the existing data product and starts over from the newly discovered date. In other words, if the startDate is changed, the sensor bot starts over.

    • Notice that the above starts a chain reaction among all indicator bots that have a data dependency with the sensor bot’s output dataset. Also, if the actual date ends up changing, all indicators that determine the starting date of the market by looking at the date discovered by the sensor bot have to discard their existing data products and start over from the new date.

Starting a Sensor Bot Instance

You do not start or stop a sensor bot instance directly. Instead, you start or stop the corresponding task.

Sensor Process Instance

A sensor process instance is a reference to the process definition of a sensor bot, as defined in a data mine.

For example, in the case of an instance of the Masters data mine Exchange Raw Data sensor bot, the bot process instance references the Historic OHLCVs process definition. Once the reference is established, the sensor process instance adopts the name of the process definition it references.

Click to learn more about sensor process instances

Adding a Sensor Process Instance Node

To add a sensor process instance, select Add Sensor Process Instance on the sensor bot instance node menu.

Starting a Sensor Process Instance

You do not start or stop a sensor process instance directly. Instead, you start or stop the corresponding task.

Indicator Bot Instance

An indicator bot instance is a reference to an indicator bot as defined in a data mine. The instance of the bot runs the defined processes and generates the defined data products.

An indicator bot is an algorithm that processes information that other bots have generated, and produces elaborate datasets for others to consume.

The indicator bot instance holds no definitions as to what the bot does. Instead, its processes instances reference the process definitions in the corresponding data mine. That is how the indicator bot instance obtains the information of what it needs to do once it is run.

Click to learn more about indicator bot instances

Adding an Indicator Bot Instance Node

To add an indicator bot instance, select Add Indicator Bot Instance on the task node menu.

Starting an Indicator Bot Instance

You do not start or stop an indicator bot instance directly. Instead, you start or stop the corresponding task.

Indicator Process Instance

An indicator process instance is a reference to the process definition of an indicator bot, as defined in a data mine.

Indicator bot instances usually require two indicator process instances. One of them references the indicator’s multi-period-market process definition and the second references the multi-period-daily process definition.

Once the reference is established, the indicator process instance adopts the name of the process definition it references.

Click to learn more about indicator process instances

Adding an Indicator Process Instance Node

To add an indicator process instance, select Add Indicator Process Instance on the indicator bot instance node menu.

Starting an Indicator Process Instance

You do not start or stop an indicator process instance directly. Instead, you start or stop the corresponding task.

Time Frames Filter

The time frame filters node allows control over which time frames are to be calculated by each indicator bot instance running on the data mining operation.

Limiting the number of time frames calculated by any given indicator to the few that may be required by a particular trading system has a significant positive impact on performance: it reduces the load on the CPU, the memory requirements, and the requirements of storage space, in proportion with the time frames you remove.

When a time frames filter is set up, a Time.Frames.json file is created by the indicator process in the corresponding output folder. This file is read by others—such as the charting system—to get the information regarding which time frames are available and which are not, to avoid reporting errors.

Click to learn more about time frames filters

Adding a Time Frames Filter Node

To add the time frames filter node, select Add Time Frames Filter on the parent node menu.

Configuring the Time Frames Filter

Select Configure on the menu to access the configuration.

{ 
"dailyTimeFrames": [ "45-min", "40-min", "30-min", "20-min", "15-min", "10-min", "05-min", "04-min", "03-min", "02-min", "01-min" ],
"marketTimeFrames": [ "24-hs", "12-hs", "08-hs", "06-hs", "04-hs", "03-hs", "02-hs", "01-hs"]
}
  • dailyTimeFrames features the time frames corresponding to the daily files type of data structure; in practical terms, the time frames below one hour.

  • marketTimeFrames features the time frames corresponding to the market files type of data structure; in practical terms, the time frames of one hour and above.

Key Reference

The key reference is a reference to an exchange account key as defined in a specific user account, in a specific exchange, on the Crypto Ecosystem hierarchy.

Usually, exchanges require autentication via your exchange account key for monetary transactions only. However, some exchanges may require autentication in other contexts as well, for instance, to retrieve information above a certain quota, or to retrieve raw trades data instead of OHLCV data.

For those reasons, exchange key references are available both in the context of data mining and trading operations, and are always attached to the corresponding task.

Forward testing and live trading sessions always require setting up key references, as that is the kind of scenario in which the user must validate with the exchange.

In all cases, the key reference node must reference a valid exchange account key from an account with the exchange, as defined in the Crypto Ecosystem hierarchy.

Click to learn more about key references

Adding a Key Reference Node

To add a key reference, select Add Key Reference on the task node menu.