Milvus
Zilliz
Star46.2KBook a DemoTry Managed Milvus
Home

dataNode-related Configurations

dataNode.dataSync.flowGraph.maxQueueLength

Description Default Value
Maximum length of task queue in flowgraph 16

dataNode.dataSync.flowGraph.maxParallelism

Description Default Value
Maximum number of tasks executed in parallel in the flowgraph 1024

dataNode.dataSync.maxParallelSyncMgrTasks

Description Default Value
The max concurrent sync task number of datanode sync mgr globally 256

dataNode.dataSync.skipMode.enable

Description Default Value
Support skip some timetick message to reduce CPU usage true

dataNode.dataSync.skipMode.skipNum

Description Default Value
Consume one for every n records skipped 4

dataNode.dataSync.skipMode.coldTime

Description Default Value
Turn on skip mode after there are only timetick msg for x seconds 60

dataNode.segment.insertBufSize

Description Default Value
  • The maximum size of each binlog file in a segment buffered in memory. Binlog files whose size exceeds this value are then flushed to MinIO or S3 service.
  • Unit: Byte
  • Setting this parameter too small causes the system to store a small amount of data too frequently. Setting it too large increases the system's demand for memory.
    		•  16777216

    dataNode.segment.deleteBufBytes

    Description Default Value
    Max buffer size in bytes to flush del for a single channel, default as 16MB 16777216

    dataNode.segment.syncPeriod

    Description Default Value
    The period to sync segments if buffer is not empty. 600

    dataNode.memory.forceSyncEnable

    Description Default Value
    Set true to force sync if memory usage is too high true

    dataNode.memory.forceSyncSegmentNum

    Description Default Value
    number of segments to sync, segments with top largest buffer will be synced. 1

    dataNode.memory.checkInterval

    Description Default Value
    the interal to check datanode memory usage, in milliseconds 3000

    dataNode.memory.forceSyncWatermark

    Description Default Value
    memory watermark for standalone, upon reaching this watermark, segments will be synced. 0.5

    dataNode.channel.workPoolSize

    Description Default Value
  • specify the size of global work pool of all channels
  • if this parameter <= 0, will set it as the maximum number of CPUs that can be executing
  • suggest to set it bigger on large collection numbers to avoid blocking
    		•  -1

    dataNode.channel.updateChannelCheckpointMaxParallel

    Description Default Value
  • specify the size of global work pool for channel checkpoint updating
  • if this parameter <= 0, will set it as 10
    		•  10

    dataNode.channel.updateChannelCheckpointInterval

    Description Default Value
    the interval duration(in seconds) for datanode to update channel checkpoint of each channel 60

    dataNode.channel.updateChannelCheckpointRPCTimeout

    Description Default Value
    timeout in seconds for UpdateChannelCheckpoint RPC call 20

    dataNode.channel.maxChannelCheckpointsPerPRC

    Description Default Value
    The maximum number of channel checkpoints per UpdateChannelCheckpoint RPC. 128

    dataNode.channel.channelCheckpointUpdateTickInSeconds

    Description Default Value
    The frequency, in seconds, at which the channel checkpoint updater executes updates. 10

    dataNode.import.maxConcurrentTaskNum

    Description Default Value
    The maximum number of import/pre-import tasks allowed to run concurrently on a datanode. 16

    dataNode.import.maxImportFileSizeInGB

    Description Default Value
    The maximum file size (in GB) for an import file, where an import file refers to either a Row-Based file or a set of Column-Based files. 16

    dataNode.import.readBufferSizeInMB

    Description Default Value
    The data block size (in MB) read from chunk manager by the datanode during import. 16

    dataNode.import.maxTaskSlotNum

    Description Default Value
    The maximum number of slots occupied by each import/pre-import task. 16

    dataNode.compaction.levelZeroBatchMemoryRatio

    Description Default Value
    The minimal memory ratio of free memory for level zero compaction executing in batch mode 0.5

    dataNode.compaction.levelZeroMaxBatchSize

    Description Default Value
    Max batch size refers to the max number of L1/L2 segments in a batch when executing L0 compaction. Default to -1, any value that is less than 1 means no limit. Valid range: >= 1. -1

    dataNode.compaction.useMergeSort

    Description Default Value
    Whether to enable mergeSort mode when performing mixCompaction. false

    dataNode.compaction.maxSegmentMergeSort

    Description Default Value
    The maximum number of segments to be merged in mergeSort mode. 30

    dataNode.compaction.lobHoleRatioThreshold

    Description Default Value
    The threshold for the ratio of unused space in existing LOB files for TEXT fields. During compaction, if this ratio is lower than the threshold, Milvus reuses the existing LOB files. If this ratio is greater than or equal to the threshold, Milvus rewrites the remaining LOB payloads into new LOB files to reclaim space. 0.3

    dataNode.gracefulStopTimeout

    Description Default Value
    seconds. force stop node without graceful stop 1800

    dataNode.slot.slotCap

    Description Default Value
    The maximum number of tasks(e.g. compaction, importing) allowed to run concurrently on a datanode 16

    dataNode.clusteringCompaction.memoryBufferRatio

    Description Default Value
    The ratio of memory buffer of clustering compaction. Data larger than threshold will be flushed to storage. 0.3

    dataNode.clusteringCompaction.workPoolSize

    Description Default Value
    worker pool size for one clustering compaction job. 8

    dataNode.bloomFilterApplyParallelFactor

    Description Default Value
    parallel factor when to apply pk to bloom filter, default to 4*CPU_CORE_NUM 4

    dataNode.storage.deltalog

    Description Default Value
    deltalog format, options: [json, parquet] json

    dataNode.text.inlineThreshold

    Description Default Value
    The size threshold, in bytes, used to choose the storage path for a TEXT field value. Values smaller than this threshold are stored inline in field data. Values greater than or equal to this threshold are stored separately as LOB payloads, and field data stores an internal reference. 65536

    dataNode.text.maxLobFileBytes

    Description Default Value
    The maximum size, in bytes, of a single LOB file that stores TEXT payloads. When a LOB file reaches this size, Milvus writes subsequent TEXT LOB payloads to another LOB file. 67108864

    dataNode.text.flushThresholdBytes

    Description Default Value
    The buffer size threshold, in bytes, that triggers Milvus to flush buffered TEXT LOB payloads from a growing segment to storage. 16777216

    dataNode.ip

    Description Default Value
    TCP/IP address of dataNode. If not specified, use the first unicastable address

    dataNode.port

    Description Default Value
    TCP port of dataNode 21124

    dataNode.grpc.serverMaxSendSize

    Description Default Value
    The maximum size of each RPC request that the dataNode can send, unit: byte 536870912

    dataNode.grpc.serverMaxRecvSize

    Description Default Value
    The maximum size of each RPC request that the dataNode can receive, unit: byte 268435456

    dataNode.grpc.clientMaxSendSize

    Description Default Value
    The maximum size of each RPC request that the clients on dataNode can send, unit: byte 268435456

    dataNode.grpc.clientMaxRecvSize

    Description Default Value
    The maximum size of each RPC request that the clients on dataNode can receive, unit: byte 536870912
    Table of contents

    Try Managed Milvus for Free

    Zilliz Cloud is hassle-free, powered by Milvus and 10x faster.

    Get Started
    Feedback

    Was this page helpful?