Spec of node joining protocols

There are at lest 3 layers in the B-tree of b-nodes that can serve the state maintenance of the search engine.

Search Engine Contract

This is simply a load balancer that orchestrates the process of joining r-nodes and c-nodes. It looks at loads of clusters and directs the incoming node to the cluster that is in need of compute.
It also maintains a network of gateways that direct incoming search transactions to clusters.

Cluster

This is simply a load balancer that orchestrates the process of joining r-nodes and c-nodes. It contains data fields:

1. load of segment_a: [total data in bytes; number of search transactions; ]
2. nodes in this segment [number of r-nodes; number of c-nodes]
3. reandomness source
4. node assignment data

When a r-node or c-node joins a network, it must contact SE-contract. SE-contract looks at the segment data and determines the segment which is most in need of compute power. Then it uses randomness to select a segment with high demand of compute power and assigns this node to the segment.

Segment

Segment S_s is a basic unit of the Fog Architecture. It contains a set of r-nodes [R_s_a] that are paralelizing the transactions that pertain to this sector.

1. Search transactions: these transctions include merkle roots of arrays returned by each of the c-nodes within the S_s
2. Crawl transactions: these transactions document performance of c-nodes on buckets in the S_s
3. Rank transactions: rank_id, schedule of computation, merkle root of the rank data structure for each of the nodes

R-node

Each segment is served by a network of R-nodes.

1. R-nodes propagate the results of search TXs submitted to them by c-nodes.
2. R-nodes submit the merged lists of URIs to the root R-node that was selected by the user to issue th esearch TX

C-node

1. Request crawl batch
2. Submit sorted URI list