NationalSecurityAgency/datawave

DataWave is a Java-based ingest and query framework that leverages Apache Accumulo to provide fast, secure access to your data. DataWave supports a wide variety of use cases, including but not limited to...

• Data fusion across structured and unstructured datasets
• Construction and analysis of distributed graphs
• Multi-tenant data architectures, with tenants having distinct security requirements and data access patterns
• Fine-grained control over data access, integrated easily with existing user-authorization services and PKI

The easiest way to get started is the DataWave Quickstart

Documentation is located here

Basic build instructions are here

How to Use this Repository

The microservices and associated utility projects are intended to be developed, versioned, and released independently. The following subdirectories contain those independently versioned modules:

contrib/datawave-utils
core/in-memory-accumulo
core/metrics-reporter
microservices/microservice-parent
microservices/microservice-service-parent
microservices/starters/audit
microservices/starters/cache
microservices/starters/cached-results
microservices/starters/datawave
microservices/starters/metadata
microservices/starters/query
microservices/starters/query-metric
microservices/services/accumulo
microservices/services/audit
microservices/services/authorization
microservices/services/config
microservices/services/dictionary
microservices/services/file-provider
microservices/services/hazelcast
microservices/services/map
microservices/services/mapreduce-query
microservices/services/modification
microservices/services/query
microservices/services/query-executor
microservices/services/query-metric

Each of those subdirectories contain a .gitrepo file that keeps track of where the code came from.

Updating one of the datawave sub-repositories

At one point we used submodules to link in a all of the sub-repositories. We have now switched to including the submodules' code directly into the main datawave repository. The git subrepo mechanism (https://github.com/ingydotnet/git-subrepo) was used to facilitate the transition. That same mechanism can be used to pull in changes from the other repositories as needed until they can be removed altogether. The original cloning of the sub repositories was done using the subrepo command as follows:

git subrepo clone <repo> <dir>

If changes need to be pulled in, then the following process can be used:

git subrepo pull <dir>

Building

The root build remains Java 11 by default. Builds run on Java 17 are supported through the java17 profile and remain backward compatible by compiling with --release 11.

When using -Pjava17, make sure Maven itself is running on a full JDK 17, not a JRE-only runtime. mvn -version should report a Java 17 runtime that includes javac. If Maven is launched with a trimmed runtime, compilation can fail with error: release version 11 not supported.

It is recommended to build the project using multiple threads. This will not build the starters, utilities, and services.

Build with Java 11:

mvn -Ddocker-release -Ddist clean install -T 1C

Build with Java 17:

mvn -Pjava17 -Ddocker-release -Ddist clean install -T 1C

If you want to build the starters, util modules, and services as well then try this:

Java 11:

mvn -Ddocker-release -Dmicroservice-docker -Ddist -Dutils -Dservices -Dstarters clean install -T 1C

Java 17:

mvn -Pjava17 -Ddocker-release -Dmicroservice-docker -Ddist -Dutils -Dservices -Dstarters clean install -T 1C

If you want to build the service APIs but not the services themselves then add -DonlyServiceApis.

When building microservices on Java 17, invoke Maven from the microservices/ reactor (or another parent that defines the java17 profile), not from an individual leaf module.

NOTE: The util modules, starters, and services are actually tagged and deployed separately. Hence the snapshot versions within those sub repos are not connected together.

DataWave Microservices

For more information about deploying the datawave quickstart and microservices, check out the Docker Readme

Key Binaries

Accumulo, Zookeeper, Hadoop, and Wildfly are the key binaries used throughout this repository. Ideally, the versions used should be the same but many of the microservices use different versions.

Additionally, there are upgrade issues with the MiniAccumuloCluster that is used in several tests so the Accumulo version is set to 2.1.2 in the root pom.xml. Because of this projects such as Quickstart must override <version.accumulo> to use a more current version.