This website is no longer actively supported. Please see the Ripple Developer Center for up-to-date documentation and other resources.

Online deletion

From Ripple Wiki
Jump to: navigation, search

Overview

Without online deletion, rippled's databases grow without bounds. To free disk space, the process must be stopped and its database files must be manually removed. Online deletion causes these databases to be pruned without disruption of service. This feature is mainly implemented in the following source file: https://github.com/ripple/rippled/blob/develop/src/ripple/app/misc/SHAMapStoreImp.cpp. Online deletion is available as of the 0.27.0 release of rippled.

Online deletion's implementation is a response to characteristics of the ledger creation and storage routines. Each ledger is stored on disk as a collection of records in a key-value store, such as RocksDB. Every new ledger is based on all of the records within the previous ledger with records added from the latest consensus round. All unchanged records remain. There is no way to distinguish within the key-value store records themselves which records are associated with a particular ledger. All records in current ledgers need to be present on disk. Any deletion must ensure that no records in the current ledgers are affected. The online deletion routine ensures that no records are removed from the current ledgers.

Since the records on disk are not organized by ledger, the mechanism to ensure that records can be safely deleted revolves around walking an entire ledger and copying all of its records into a new key-value database. This is a time-consuming operation, and cannot be done within every ledger close period. Therefore, the ledger copying is handled by an independent thread and performed at configurable intervals. Every online deletion event ensures that all records necessary for continued activity are copied to a new key-value store, keeps all of the data acquired within the previous interval, and deletes data prior to that. For finer-grained control, there is an optional mechanism to require an administrative RPC call to activate online deletion events.

Records stored in internal `SQLite` tables are deleted based on ledger index, and correspond to the records deleted from the key-value store.

Deployment

Online deletion is configured with two parameters within the [node_db] section. online_delete takes an integer value that defines the number of ledgers between online deletion events. It must be at least 256 to maintain stability of the Ripple peer-to-peer network. An optional parameter, advisory_delete, requires an administrative command to be submitted to trigger deletion. This does not over-ride the online_delete interval. advisory_delete is enabled if this value is set and greater than 0.

Online delete keeps two distinct key-value databases on disk at all times, and does not reference any pre-existing key-value store. Therefore, a pre-existing key-value store should be removed prior to implementation. The ledger and transaction SQLite databases should also be removed. Other databases can be left alone, but all ledger and transactional data will start from scratch.

The online deletion interval must be greater than or equal to the [ledger_history] section. rippled will not start if this condition is not met. Also, [fetch_depth] must be less than or equal to the online delete setting. If configured differently, rippled will force [fetch_depth] to equal that of the online deletion interval and start normally.

  • If running, stop rippled.
  • Edit the rippled.cfg [node_db] section to include the desired online deletion parameters, such as:
[node_db]
type=rocksdb
path=/var/db/rippled/db/nodestore
online_delete=1024
advisory_delete=1

[ledger_history]
1024
  • Remove existing data:
sudo rm /var/db/rippled/db/ledger.db* /var/db/rippled/db/transaction.db* /var/db/rippled/db/nodestore/*
  • Start rippled.

Operation

Upon every online deletion event, log messages are recorded describing the progress of the activity. These log messages are written to the [debug_logfile] and contain the text SHAMapStore.

Online deletion has the potential to decrease performance of rippled based on increased I/O activity and because of cache clearing. Therefore, the health of rippled is checked frequently during online deletion events. If server_state becomes anything other than full or if validated ledger age is at least 60 seconds, then the online deletion event will stop. In this case, it will restart at the next validated ledger close interval. In this way, online deletion runs safely in the background to minimize impact. If an online deletion event is aborted in this way, a message in the warning category will be written to [debug_logfile]. Filter the file with the following: SHAMapStore::WRN. Likewise, if rippled shuts down or if the server crashes during online deletion, online deletion will begin again after startup and the next validated ledger.

The amount of data retained at all times will range based on ledger quantity between the minimum of the configured online_delete interval, and the total data acquired between three online deletion events. If no advisory_delete is set, then the upper limit will be roughly twice the online_delete interval. If, on the other hand, advisory_delete is set and online_deletion is triggered once per day, then the amount of data online will range between one and two full days' worth of data. The complete_ledgers field in the server_info RPC call indicates the precise range of ledgers at any time. Data size of the ledger and transaction databases in SQLite will steadily grow over time, but at a relatively slow rate. These SQLite files on disk will not shrink after online deletion, but space within the files will be made available for new data. These files will only grow as ledger size grows. Upon restart, rippled will perform a SQLite VACUUM operation to minimize the size of these files.

Online delete state is retained in an SQLite database file called state.db in the [database_path] directory. Both key-value database locations, the last rotated ledger index, and the advisory_delete ledger, if any, are all stored in this database. If the key-value databases on disk do not match those listed in state.db, then rippled will emit an error to the console and not start. In this case, some type of corruption has occurred. It is necessary to remove the state.db file and the contents of the path directory in [node_db] before restarting rippled. rippled will start afresh with no key-value store data. If desired, make backups of state.db and the [node_db] path prior to performing this activity.

Examples

Using advisory_delete

The administrative RPC call required to trigger online deletion events is called can_delete. Using the above configuration, to trigger an online deletion event at five minutes past 1AM every morning, put the following in the crontab of a user able to run rippled locally:

5 1 * * * </path/to/rippled> --conf <path/to/rippled.cfg> can_delete now

To tell rippled that it may delete any ledger less than or equal to 11,500,000, use the following command line command:

</path/to/rippled> --conf </path/to/rippled.cfg> can_delete 11500000

To disable online deletion (until this setting is changed manually), use the following command line command:

</path/to/rippled> --conf </path/to/rippled.cfg> can_delete never

To no longer require the can_delete administrative RPC call for online deletion:

</path/to/rippled> --conf </path/to/rippled.cfg> can_delete always

Reminder: Regardless of advisory_delete and can_delete settings, online deletion will only occur after the online_delete interval since the last deletion (or since initial rippled startup).

Online Deletion Process - Without advisory_delete

Suppose the current ledger is 1000, and the rippled.cfg has the following conditions :

[ledger_history]
256
[node_db]
.
.
.
online_delete = 500
advisory_delete = 0

Online Deletion Process :

  1. Fetch ledgers 744 - 1000
    • Satisfies [ledger_history]
    • Ledgers are stored in the writable database.
  2. Progress ledgers to 1500
    • Do not delete ledgers, but shift the current writable database to the read-only database.
    • The read-only database contains 744-1500
    • The new current writable database has everything 1501+
  3. Progress ledgers to 2000
    • Delete ledgers 744-1500 (the read-only database).
    • Create a new writable database.
    • The writable database (1501-2000) becomes read-only.
    • The new current writable database has everything 2001+
  4. Progress ledgers to 2500

Online Deletion Process - With advisory_delete

Suppose the current ledger is 1000, and the rippled.cfg has the following conditions :

[ledger_history]
256
[node_db]
.
.
.
online_delete = 500
advisory_delete = 1

Online Deletion Process :

  1. Synchronize to network for the first time ever at ledger 1000
    • Set marker for rotation at ledger 1000
    • Fetch ledgers 744 - 1000 (Satisfies [ledger_history])
    • Ledgers are stored in the writable database.
  2. Progress ledgers to 1500
    • Do not delete ledgers. Do not rotate ledgers until can_delete has been called with a minimum value of 1000, "now", or "always".
    • If not, the read-only database stays empty.
    • The current writable database contains 744-1500
  3. Progress ledgers to 2000
    • Do not delete ledgers. Do not rotate ledgers until can_delete has been called as described previously.
    • If not, the read-only database stays empty
    • The current writable database contains 744-2000
  4. can_delete is called with parameter 1200 at ledger 2149
</path/to/rippled> --conf </path/to/rippled.cfg> can_delete 1200
  • The online deletion routine activates at the next validated ledger, 2150.
  • The readable database is discarded (so far has been empty)
  • The writable database with ledgers 744-2150 becomes read-only.
  • A new writable database is created and will store ledgers 2151+.
  • The marker for last rotation is now 2151.

The next rotation will occur when both another 500 ledgers pass by and can_delete is called with at least a value of 2150.

Online Deletion Process - With advisory delete scheduled daily

Set the following in cron to execute at 12:05AM:

5 0 * * * /usr/sbin/rippled --conf /etc/rippled/rippled.cfg can_delete now

Suppose the current ledger is 1000, and the rippled.cfg has the following conditions :

[ledger_history]
256
[node_db]
.
.
.
online_delete = 500
advisory_delete = 1

Set the following in cron to execute at 12:05AM:

5 0 * * * /usr/sbin/rippled --conf /etc/rippled/rippled.cfg can_delete now

Online Deletion Process:

  1. Synchronize to network for the first time ever at ledger 1000
    • Set marker for rotation at ledger 1000
    • Fetch ledgers 744 - 1000 (Satisfies [ledger_history])
    • Ledgers are stored in the writable database until at least 500 ledgers have been processed and until can_delete is called appropriately.
    • There are approximately 18,000 closed ledgers in a full day.
    • At 12:05AM, cron executes the "can_delete now" RPC call. Assume the next validated ledger is 10000.
    • The current writable database contains ledgers 744 - 10000. It becomes read-only, and a new writable database is created.
    • The marker for last rotation is set to 10000.
    • The next deletion routine will occur when at least 500 ledgers go by, and can_delete is called with at least the value of 10000, or "now", or "always".
    • At 12:05AM the next morning, cron executes "can delete now". The next validated ledger is 28000, and databases are created, shifted, and deleted as before.


Online Deletion Process - With advisory_delete text commands

  • never: deletion routine will not execute at all unless overridden with another can_delete call, or rippled is reconfigured without advisory_delete set and restarted.
  • always: deletion routine will occur at every interval configured with the online_delete command, and will not require future can_delete calls. Can be overriden the same way as the "never" argument to can_delete.
  • now: deletion routine will occur as of the next validated interval that also equals or exceeds the online_delete interval since the last execution of the routine. No further deletions will occur unless triggered again by can_delete.