gchq/stroom

Stroom is a data processing, storage and analysis platform. It is scalable - just add more CPUs / servers for greater throughput. It is suitable for processing high volume data such as system logs, to provide valuable insights into IT performance and usage.

Stroom provides a number of powerful capabilities:

• Data ingest. Receive and store large volumes of data such as native format logs. Ingested data is always available in its raw form.
• Data transformation pipelines. Create sequences of XSL and text operations, in order to normalise or export data in any format. It is possible to enrich data using lookups and reference data.
• Integrated transformation development. Easily add new data formats and debug the transformations if they don't work as expected.
• Scalable Search. Create multiple indexes with different retention periods. These can be sharded across your cluster.
• Dashboards. Run queries against your indexes or statistics and view the results within custom visualisations.
• Statistics. Record counts or values of items over time, providing answers to questions such as "how many times has a specific machine provided data in the last hour/day/month?"

Get Stroom

To run Stroom and Stroom-Proxy in Docker do the following:

# Download and extract Stroom v7.10 stack
bash <(curl -s https://gchq.github.io/stroom-resources/v7.10/get_stroom.sh)

# Navigate into the new stack directory
cd stroom_core_test/stroom_core_test*

# Start the stack
./start.sh

For more details on the commands above and any prerequisites see Single Node Docker Installation.

For the releases of the core Stroom and Stroom-Proxy products, see Stroom releases.

For the releases of the docker application stacks, see Stroom-Resources releases.

Documentation

The Stroom application spans several repositories but we've bundled all the documentation into one place.

See Stroom Documentation.

Contributing

Contributing to Stroom can come in the following forms:

• Raising issues for bugs or feature requests in Stroom.
• Making code contributions to fix bugs or adding new features.
• Developing new Stroom content (see stroom-content).
• Adding to or updating the documentation.

If you'd like to make a contribution to Stroom then the details for doing all of that are in CONTRIBUTING.md.

If you would like to contribute to Stroom's documentation then see here. The documentation repository can be found at github.com/gchq/stroom-docs.

GitHub Repositories

Stroom and its associated libraries, services and content span several GitHub repositories:

• stroom - The core Stroom application and proxy.
• stroom-clients - Various client libraries for sending logs to Stroom.
• stroom-content - Packaged content packs for import into Stroom.
• stroom-docs - Documentation for the Stroom family of products.
• stroom-headless - An example of how to run Stroom in headless mode from the command line.
• stroom-resources - Configuration for orchestrating stroom in docker containers and released docker stacks.
• stroom-visualisations-dev - A set of visualisations for use in Stroom.
• event-logging-schema - An XML Schema for describing auditable events.
• event-logging - A JAXB API for the event-logging XML Schema.