## hashline128

A lightweight, secure, and Unix-compliant filter for line-based hashing of data streams.

## The MurmurHash3 (128-bit) Algorithm

The tool utilizes the 128-bit variant of the MurmurHash3 algorithm, an extremely fast, non-cryptographic hash function optimized specifically for the performance of modern 64-bit processors, reaching RAM speed limits.

* Mathematical Principle: The algorithm processes data in large blocks (two parallel 64-bit halves for 128-bit generation) and uses mathematical operations such as multiplications, bit rotations (rotates), and XOR operations with predefined, mathematically optimized constants. At the end of the data stream, the internal states are mixed (finalization) and merged into a single 128-bit hash value.
* Difference from FNV-1a: While FNV-1a processes data strictly byte by byte, MurmurHash3 internally utilizes instruction-level parallelism (ILP). This makes MurmurHash3 many times faster for larger volumes of data and optimally utilizes the CPU pipeline.
* Comparison with djb2 and xxHash: MurmurHash3 outperforms older functions like djb2 in processing speed while maintaining excellent hash distribution. It passes all strict statistical tests of the SMHasher test suite and, due to its 128-bit output, offers an astronomically low collision probability, enabling secure deployment in hash tables even with extremely large datasets.
* Application Areas: Optimized for high-speed checksums, real-time data streams, file comparisons, and high-performance in-memory hash tables (e.g., in databases or routers).
* Limitation: Since the function is not cryptographically secure, it is vulnerable to targeted collision attacks (hash-flooding / DoS attacks) if adversaries can fully control the input data.

## Buffer Behavior (Fail-Fast)

The program processes lines up to a fixed limit of 64 KB (BUFSIZE / 65535 bytes). If a line exceeds this limit, the tool follows the fail-fast principle:

* It terminates the data stream in a controlled manner with exit code 0.
* No memory allocation beyond the limit, no silent data manipulation, and no program crashes occur.
* A separate error code is not issued, as this is a defined and documented program behavior. The overflow can be identified within the data stream itself, as the final line will exactly match the maximum buffer size (important for traceability in log files).

## Architecture and Unix Philosophy

This restrictive behavior aligns with the Unix philosophy ("Do one thing and do it well"). By avoiding features like dynamic memory management (realloc), the tool provides specific advantages:

* Performance: No runtime overhead caused by dynamic memory allocation.
* Security: The static buffer structurally eliminates heap exploits.
* Compactness: The small code size is ideally suited for static linking with musl-gcc.
* Platform Security: Software-level hardening using -fstack-protector-strong and -static-pie mitigates remaining risks.

## Pipeline Integration

Adjustments for non-standard data streams must be handled by upstream tools within the pipeline.

## Preprocessing Options:

   1. Truncating overly long lines:

   An upstream cut command prevents the program from terminating and protects system resources:

  ... | cut -c-65532 | hashline128

   2. Enforcing line breaks:

   An upstream fold command actively wraps overly long data streams after a specific number of characters. This prevents the program from terminating but alters the structure of the input data for the hash calculation:

  ... | fold -w 65532 | hashline128

   3. Fragmentation with a delimiter:

   An upstream awk command cleanly splits overly long lines and appends a delimiter (\036) so that processing can continue. The fragments must be manually merged at a later stage:

  ... | awk '{while(length($0)>65535){printf "%s\036\n",substr($0,1,65532);$0=substr($0,65533)}print}' | hashline128

## Adjusting the Buffer Size

The 64 KB limit is fixed to keep the memory footprint low. Since the source code is in the public domain, users can manually adjust the BUFSIZE constant in the source code if needed and compile their own binary.