Network definition

YaCy peer-to-peer network is completely decentralized and also does not require a single central server for the network to clamp up.

Network-Bootstrapping

Of course, a 'new' peer must know how to contact the other peers, for that is there so-called seed list. That, generated by a peer in the network, is used for the Network-Bootstrapping, but any participant of the network can generate seed-list. A peer that creates such a file may call itself a 'principal peer' and there can be several of them. The network is therefore defined by very specific peers, but these cannot be viewed as 'centers' because, in principle, every peer can be principal.

Now you still may ask yourself where a new peer finds the principal peers of the network: these are stored in the network definition file of the network, that you want to join. A new YaCy peer can basically connect to any network, but that doesn't mean you must set something complicated: every installation includes a network definition file of default network called 'freeworld'. That is located in defaults/yacy.network.freeworld.unit and is set by the attribute network.unit.definition in defaults/yacy.init. You don't have to adjust or change anything in that configuration file, everything neccessary for the "freeworld" network is already set.

However, every YaCy user can define their own network, and this article covers in detail how that works.

The processes that happen during bootstrapping are:

  • look-up the network.unit.definition attribute in defaults/yacy.init. The value of this attribute is either a path to a file or URL of a network definition. In case of Freeworld, the values are stored in defaults/yacy.network.freeworld.unit
  • read the defaults/yacy.network.freeworld.unit; there are attributes network.unit.bootstrap.seedlist0, network.unit.bootstrap.seedlist1 etc., which contain the URLs of the seed lists
  • the files from the values of network.unit.bootstrap.seedlist0 (..) are read. They contain so-called Peer-Seeds. These are brief information about the peers, which states which IP they have and a lot more like name, index size and so-called peer news
  • some of the seeds accumulated in this way are used to grow the seed in question to send a so-called 'peer ping'. In this ping one's own peer propagates its own seed information, which is stored by the pinged peer and sent to other peers.
  • in response to the ping, the pinged peer sends the latest information from other peers by sending their seeds. On other peers then find out about their own peer in the same way the peers pinged in this way pass on information with the new seed.

After the seed lists have been loaded once, a peer can even completely find its way back to the network without loading the peer lists.

Definition of the YaCy network with yacy.network.unit

In peer-to-peer mode, YaCy creates a network cluster of YaCy peers defined for a specific domain of web-index. By default, this search network is the public YaCy network, whose domain is the public Internet. The network is defined through the bootstrapping, and all peers within the network must have the same bootstrapping information with the properties of the network. These information are stored in yacy.network.unit.

The yacy.network.unit file is included in every YaCy instance from the default settings (yacy.init) with the property network.unit.definition, which defines the following properties, among others:

network.unit.name = freeworld
network.unit.description = Public YaCy Community
network.unit.domain = global
network.unit.dhtredundancy.junior = 1
network.unit.dhtredundancy.senior = 3
network.unit.bootstrap.seedlist0 = http://www.yacy.net/yacy/seed.txt
network.unit.update.location0 = https://download.yacy.net/

At the YaCy startup, the network is set up as follows:

  1. the yacy.init is loaded: the property network.unit.definition in yacy.init denotes yacy.network.unit as the network definition.
  2. yacy.network.unit is loaded: the property network.unit.bootstrap.seedlist0 in yacy.network.unit is set to "http://www.yacy.net/yacy/seed.txt", for a list of seeds from the network
  3. http://www.yacy.net/yacy/seed.txt is loaded and the seeds in it are loaded into the seed DB.
  4. The seeds contain information about the last known peer addresses.

The peers of this network all use the name mentioned in yacy.network.unit (network.unit.name) to identify themself as a participant in the same network. The property network.unit.description is just a free definable text that is displayed in the network graphic, for example.

A very important information is the web domain which the network indexes. The associated property is called network.unit.domain. The domain and can take the following values:

  • global: only URLs that are freely accessible are in the index accepted.
  • local: only URLs that are accessible on an intranet will be accepted. This is useful, for example, when indexing a intranets.
  • any: both local and global addresses are accepted.

Another network-related setting is the "redundancy factor" or the number indicating how many copies of index are distributed within the DHT. In a public network, the availability of a peer cannot be ensured and therefore the redundancy factor is 3 (network.unit.dhtredundancy.senior). In a network with high availability, this factor can be set to 1.

If all peers in a network are administered by a single person you may want an automatic update of all the peers to take place. To do this, a download location must be defined, and a network operator can specify own location to be able to control the version for the automatic update. To do this, the property network.unit.update.location0 can contain the URL of page that contains links to releases. See autoupdate for details.

You can provide additional alternative addresses both to network.unit.bootstrap.seedlist0 and network.unit.update.location0; you can simply add additional properties named like that, with increased sequence number.

Creation of your own YaCy network

The network definition must be the same for all participants in a network, and this is achieved - in the standard case - by including that in the release.

A YaCy network operator may be interested in changing the network definition after the network has been set up for all peers, for example for advanced security settings for the network. Hence the setting the network.unit.definition in yacy.init is possible also via URL, which we use in the following example.

The construction of a new network consists of two major steps, first is the definition of the first peer of the new network and then in the deployment of the other peers assigned to the first peer.

Configuration of the first peer of a new network

The steps are:

  • Editing the yacy.network.unit, for example to index both local and global websites, but with the update address of new releases from the global network:
network.unit.name = mynet
network.unit.description = My first very own YaCy network
network.unit.domain = any
network.unit.dhtredundancy.junior = 1
network.unit.dhtredundancy.senior = 1
network.unit.bootstrap.seedlist0 = http://www.meindomain.invalid/yacy.myseedlist
network.unit.update.location0 = https://download.yacy.net/

network.unit.definition = http://www.meindomain.invalid/yacy.mynetdef

This is only necessary for the initial installation, further updates of the peers must remain constant and cannot be set again.

  • Now, the first peer of the network can be started. For the second peer to find it, it must know its IP. YaCy usually uses peer ping to distinguish itself from another peer to have his public IP named. At the first peer and a new network, this is not possible because there is no other peer yet that could respond to a peer ping. Instead, your own public IP must be assigned via the menu item http://<address_of_first_peer>/Settings_p.html?page=ServerAccess , and configure the IP in StaticIP setting.

  • The first peer must operate as a principal peer, i.e. it must be able to create a seedlist so that the newly started peers can find the first peer

    • Under http://<address_of_first_peer>/Settings_p.html?page=seed the upload address for the seed list can be defined. This was already entered in yacy.mynetdef and was http://www.meindomain.invalid/yacy.myseedlist
    • check whether the peer reaches principal status, i.e. it was able to create a seed list and complete the upload.

Configuration of the participating peers of the new network

Once the first peer is running, additional peers can be added. They load the seedlist and contact the principal-peer, which provides the new information again through a seedlist-upload to the newly connected peer.

To ensure that new peers can automatically access the new network, you can make a special YaCy release with settings of the new network. To become a new peer, correctly configured to participate in the new network, it can also be configured with normal, unchanged release and updated without loosing network membership. The steps to define the Special releases are:

  • Configuring the network definition in yacy.init:
network.unit.definition = http://www.meindomain.invalid/yacy.mynetdef
  • It also makes sense to set-up the automatic updates of the peers:
update.process = auto
  • To avoid neccessity of entering the passwords via setup menu during mass installations, it is recommended to set a default password in yacy.init, according to the following example:
adminAccount=admin:myS3cr3tPa55w0rd
  • Once the yacy.init is fully configured, you can create your own Bootstrap-release for your own network, simply by packing the yacy directory:
tar cf yacy_mynet.tar yacy

For a mass deployment, all you have to do to install YaCy on the network's computers, is to distribute and unpack the file yacy_mynet.tar.

To ensure a permanent availability of the YaCy installation, it's recommended to define a cron job that regularly restarts installed peer. For example, using following entries in /etc/crontab file:

0 0 * * * yacyuser /home/yacyuser/yacy/stopYACY.sh
2 0 * * * yacyuser /home/yacyuser/yacy/killYACY.sh
4 0 * * * yacyuser /home/yacyuser/yacy/startYACY.sh -l

or, simply:

0 3 * * * yacyuser /home/yacyuser/yacy/restartYACY.sh

provided that YaCy runs under the user 'yacyuser' and that YaCy directory is located in the yacyuser home directory.

For more network settings see also: yacy.network.readme.

Converted and translated from German from http://wiki.yacy.net/index.php?title=De:Netzdefinition. May be outdated.