## xxhashline64

A lightweight, secure, and Unix-compliant filter for line-based hash generation from data streams.

## The xxHash (XXH64) Algorithm

The tool utilizes the XXH64 algorithm, an extremely fast, non-cryptographic hash function optimized specifically for the performance of modern 64-bit processors, reaching RAM speed limits.

* Mathematical Principle: The algorithm processes data in large blocks (4 parallel 64-bit lanes) and utilizes mathematical operations such as multiplications, bit rotations (rotates), and XOR operations with predefined, mathematically optimized primes. At the end of the data stream, the lanes are merged into a single 64-bit hash value (finalization).
* Difference from FNV-1a: While FNV-1a processes data strictly byte by byte, XXH64 internally utilizes Instruction-Level Parallelism (ILP). As a result, XXH64 is many times faster for larger volumes of data and optimally utilizes the CPU pipeline.
* Comparison with djb2 and MurmurHash3: XXH64 outperforms djb2 and MurmurHash in processing speed while maintaining excellent hash value distribution. It passes all rigorous statistical tests of the SMHasher test suite (no known systematic collisions or flaws in the avalanche effect).
* Fields of Application: Optimized for high-speed checksums, real-time data streams, file comparisons, and high-performance in-memory hash tables.
* Limitation: Since the function is not cryptographically secure, it is vulnerable to targeted collision attacks (hash flooding / DoS attacks) if adversaries can fully control the input data.

## Buffer Behavior (Fail-Fast)

The program processes lines up to a fixed limit of 64 KB (BUFSIZE / 65535 bytes). If a line exceeds this limit, the tool behaves according to the fail-fast principle:

* It terminates the data stream in a controlled manner with exit code 0.
* No memory allocation beyond the limit, no silent data manipulation, and no program crash take absence.
* A separate error code is not issued, as this represents a defined, documented program behavior. The overrun is identifiable within the data stream itself, as the final line corresponds exactly to the maximum buffer size (critical for traceability in log files).

## Architecture and Unix Philosophy

This restrictive behavior adheres to the Unix philosophy ("Do one thing and do it well"). By eschewing features like dynamic memory management (realloc), the tool provides distinct advantages:

* Performance: No runtime overhead from dynamic memory allocation.
* Security: The static buffer structurally eliminates heap exploits.
* Compactness: The small code size is ideally suited for static linking with musl-gcc.
* Platform Security: Software-side hardening via -fstack-protector-strong and -static-pie mitigates remaining risks.

## Pipeline Integration

Adjustments to non-standard data streams must be resolved by upstream tools within the pipeline.

## Options for Preprocessing:

   1. Truncating overly long lines:
   An upstream cut prevents program termination and protects system resources:

 ... | cut -c-65532 | xxhashline64

   2. Enforcing line breaks:
   An upstream fold actively wraps overly long data streams after a specific number of characters. This prevents the program from terminating, but alters the structure of the input data for hash calculation:

 ... | fold -w 65532 | xxhashline64

   3. Fragmentation with delimiters:
   An upstream awk command cleanly breaks overly long lines including a delimiter (\036) so that processing can continue. The fragments must be merged manually at a later stage:

 ... | awk '{while(length($0)>65535){printf "%s\036\n",substr($0,1,65532);$0=substr($0,65533)}print}' | xxhashline64

## Adjusting the Buffer Size

The 64 KB limit is a fixed choice to keep memory requirements low. Since the source code is in the public domain, users can manually adjust the BUFSIZE constant in the source code if necessary and compile their own binary.