#Splunk dedup command how to

For example, the rex command is streaming. When the search processing moves to the search head, it can't be moved back to the indexer.ĭistributable streaming commands can be applied to subsets of indexed data in a parallel manner.

If any one of the commands before the distributable streaming command must be run on the search head, the remaining commands in the search must be run on the search head.

If all of the commands before the distributable streaming command can be run on the indexer, the distributable streaming command is run on the indexer.

The other commands in a search determine if the distributable streaming command is run on the indexer: A distributable streaming command is a command that can be run on the indexer, which improves processing time. For distributable streaming, the order of the events does not matter.

See Data processing commands.Ī streaming command operates on each event returned by a search. The sort command is an example of a data processing command. These commands are not transforming, not distributable, not streaming, and not orchestrating.

After you run a transforming command, you can't run a command that expects events as an input.ĭata processing commands are non-streaming commands that require the entire dataset before the command can run. The events used to calculate those results are no longer available. For example the stats command outputs a table of calculated results. However, transforming commands do not output events. For example, when you run the sort command, the input is events and the output is events in the sort order you specify. When a command is run it outputs either events or results, based on the type of command. The following table describes the processing differences between some of the types of commands.

#Splunk dedup command how to

This requires a lot of data movement and a loss of parallelism.įor information on how to mitigate the cost of non-streaming commands, see Write better searches in this manual. Non-streaming commands force the entire set of events to the search head. Other examples of non-streaming commands include dedup (in some modes), stats, and top. Essentially one event in and one (or no) event out.įor example, before the sort command can begin to sort the events, the entire set of events must be received by the sort command. For example a command can be streaming and also generating.įor a complete list of commands that are in each type, see Command types in the Search Reference.Ī streaming command operates on each event as it is returned by a search. Other commands can fit into multiple categorizations. The stats command is an example of a command that fits only into the transforming categorization.

Some commands fit into only one categorization. These categorizations are not mutually exclusive. There are six broad categorizations for almost all of the search commands: This topic explains what these terms mean and lists the commands that fall into each category. As you learn about Splunk SPL, you might hear the terms streaming, generating, transforming, orchestrating, and data processing used to describe the types of search commands.