P10 Protocol

Revision as of 12:57, 9 July 2010

This page needs an update. Information posted here has been viewed as incorrect, incomplete, or out of date. Anyone is welcome to correct these flaws if this page has not been locked. Otherwise, contact a Support Team member and give them notice of this issue. Thank you.

This document is based on Undernet's P10 protocol specification as of ircu2.10.11. It aims to be a practical guide for implementing and maintaining the protocol, with DareNET's extensions, not just a reference manual; therefore, it can be considered a "work in progress," continually being updated.

Terminology

Definitions of some commonly used terms in this document.

• Byte - A unit of 8 bits of data.

• Character - One byte, notated as a decimal number in the range 0-255, or a printable ASCII character (example: 65, 'A').

• Char - Character.

• String - A sequence of bytes.

• Space - ASCII 32 ($20).

• CR - Carriage return. '\r' (ASCII 13)

• LF - Line feed (also known as 'new line'). '\n' (ASCII 10)

• CRLF - Carriage return + Line feed (<CRLF>).

• NULL - Character 0.

• EOL - End of Line (line termination).

• Parser - The implementation which receives and processes the stream.

• Generate - Sending data which has not been received, as opposed to passing data on which has been received.

• TS - Timestamp. An ASCII decimal notation of a date+time since Unix epoch.

• Nick - Refers to a nickname.

• Numnick = Refers to a client's numeric nickname.

"Must", "must not", "should", "may" are as described in RFC2119. Interpret "disallowed" as "must not."

Hexadecimal numbers in this document use pascal notation: a $ prefix. The number of hex digits (nibbles) represents the size of the data; for example, a byte can be anything between $00 and $ff.

Stream of Data, Lines, Line Termination

P10 is a 'text' protocol, that is, it is human readable and writable. Communication between two servers is done by means of a stream of bytes. The stream must be connected, reliable and ordered. So, the definition of the stream would be as follows:

<line><EOL><line><EOL> ... <junk>

Line termination (EOL)

When sending, line termination may be either <CRLF> or <LF>. It must not be anything else.

The parser must accept <LF> and <CRLF> as line termination. It may accept any other sequence of <CR> and <LF> as EOL. It must not parse anything else as line termination.

Line

A line (also commonly referred to as a "message") is defined as a sequence of characters, minimum length being 1 byte, maximum length 510 bytes, not including the EOL (which makes a total of 512 bytes). If a parser encounters a line with a length of 0 bytes, it must be silently ignored, and it must not do anything else. A line which is longer than the maximum length is disallowed.

NULL, CR and LF are disallowed in a line, any other character is allowed.

A parser can encounter a line which contains a NULL character. It may terminate the line at the first NULL character (removing anything after and including the first NULL character from the line).

Junk

Any data between the last EOL and the end of the stream. It must not be parsed as a line.

P10 Base64

The P10 protocol uses a modified base64 notation for numeric nicks, and for the IP parameter in the NICK message. It uses the following set of 64 characters, in the sequence from 0 to 63 (henceforth referred to as the P10 base64 characters):

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789[]

A P10 base64 string is a sequence of P10 base64 characters, with a minimum length of 1 character. It must not contain any other character. If the string has a length of more than one character, the string begins with the most significant character and ends with the least significant character.

P10 base64 strings are case sensitive.

Numerics

The P10 protocol uses a scheme of "numerics" to uniquely identify a client or server within the network. Each server has its own unique numeric (0 -> 4095) and each client has its own numeric within that server (0 -> 262,143).

These numerics are encoded into a base64 string to maintain human readable data flow, and to reduce the size of messages.

Server numerics consist of 2 characters, with the minimum, 0, being represented by "AA", and the maximum, 4095, being represented by "]]". Client numerics are 3 characters in length, with the minimum, 0, being represented by "AAA", and the maximum, 262,143, being represented by "]]]". The unique identifier (known as a numnick) of a client on the network consists of a combination of both the server and client numeric in the format SSCCC - a total of 5 base64 characters.

A server has a maximum client numeric, which is sent in the SERVER message. A client numeric on a server, ANDed with the maximum server numeric, must be unique. For example, if the server YY's max client numeric is YYA[[ (4095), clients YYBXX and YYCXX cannot exist at the same time. Doing this would result in a "numeric collision", which is a protocol violation. However, such numerics which occupy the same "slot" are not identical - a message sent to use YYBXX in this example must not reach user YYCXX.

Taken what we know from above, let's go through an example. Consider a server "peanuts.darenet.org" which as a numeric of 2, translating to "AC" in base64. On this server exists a client, whom has been allocated the numeric 63 (which translates to "AA]" in base64); therefore, the unique identifier of this client on the network is "ACAA]". From this, we can determine which server the message came from, as well as the client who sent it.

The P10 protocol defines two types of numerics: short numerics and extended numerics. For the purpose of this document, we have chosen to use extended numerics, as implemented by Undernet, DareNET and most other networks utilizing the P10 protocol.

Syntax of a line/message

The source, command, and parameters are separated by spaces.

<source> <command> [<parameters>]

One must send only the short command token. One must parse both the short command token and its long unabbreviated name, and if one does, they must be considered equivalent; for example N = NICK. So if I say "receives a NICK message" it may actually be a N token.

Command tokens are uppercase. One must not send lowercase command tokens. One may parse them.

If source begins with a colon, it (except for the colon) is the name; otherwise, it is a numeric. One must only send messages with a numeric source prefix, except for the initial PASS and SERVER command, which are not prefixed.

If the source does not exist, if the command is SQUIT or KILL (or their respective short token), the message must be parsed anyway, with the directly linked server from which the message came as the source; otherwise, the message must be ignored.

If the source exists but the message comes from the wrong direction, it must be ignored. The direction of a source is defined as the link over which the corresponding message was received, that originally introduced the source. A link is the connection with another server.

A message may have up to 15 parameters. Parameters are separated by spaces.

The last parameter may be prefixed by a colon; this allows the last parameter to have spaces, or to have a length of 0 characters. For example: <source> <command> <param1> <paramN> :<last parameter>. A parser must be able to parse messages with colon prefixed last parameter, and without. For example, parameters "a b c" and "a b :c" are equivalent.

Command Token Table

The below table lists all acceptable messages, along with their relevant "token", which is used in server<>server communication. The aim of tokenization is to reduce the bandwidth used during network communication by reducing the total length of common message identifiers.

Registration and Synchronization

Server Registration and Authentication

After a TCP connection has been established, the server initially introduces itself via a "PASS" message as follows:

PASS :<password>

<password> is simply a plain text password that is compared with the password present in the destination servers configuration file, and is used to confirm credentials after "SERVER" message has been received, as follows:

SERVER <server_name> <hop_count> <boot_time> <link_time> <protocol> <numeric/maxconn> :<description>

For example:

1      2                   3 4         5         6   7     8
SERVER peanuts.darenet.org 1 933022556 947908144 J10 AA]]] :[127.0.0.1] A DareNET Client Server.

Notes:

1. The SERVER message indicates this connection wishes to introduce a new server to the network.
2. <serv_name> - The name of the server you are introducing, must be a valid server name.
3. <hop_count> - The hop count of the server you are introducing; this is always 1 when you are introducing yourself.
4. <boot_time> - The epoch timestamp specifying when the ircd was started.
5. <link_time> - The epoch timestamp specifying the time the server initiated the link to the network.
6. <protocol> - The protocol identifier for this server.
   1. If the server being introduced has not yet successfully synced its database with the network (i.e., completed its net.burst), then the protocol token should be prefixed with a J, instead of a P (e.g., J10) to indicate it is currently still joining the network.
   2. The protocol token should always be JXX when the server is introducing itself.
7. The numeric and maximum connections identifier for this server.
   1. This token is formatted exactly the same as a client numeric is formatted. The first 2 characters identify the server's numeric, whilst in this situation, the final 3 characters define the maximum number of clients that this server can hold (and more importantly, the maximum number of numerics it will generate). This is always one less than a power of two, because the server uses this as a bitmask. A server can give out a higher numeric than this; however, it will be ANDed with this number to find its entry slot. The reason for this is so a server which is near the maximum number of clients can give out more numerics than it's using to prevent a new client getting a numeric that was used only seconds ago, and maybe get messages destined to the old user.
   2. The example "AA]]]" shows that this is a server with numeric 0, which will generate client numerics up to 262,143.
8. <description> - This final parameter simply consists of a textual description of the server prefixed by a colon. This is displayed in a clients WHOIS line, as well as in the LINKS reply. By convention, if this is a leaf server, it contains the servers IP in square brackets at the beginning of the string.

Boot timestamp

As mentioned above, this is an epoch timestamp indicating when the ircd was started. It can be seen as the age of the net/net fragment. If a boot TS in a received SERVER message is younger than OLDEST_TS (780000000), and older than the existing boot TS, the existing boot TS is overwritten with the received boot TS. The boot TS can be used for making decisions.

Link timestamp

As mentioned above, this is a n epoch timestamp specifying the time the server initiated the link to the network. Both ends of a link have the same link TS for that link: it is the link TS parameter in the SERVER message sent from the accepting server to the connecting server. When a server link is established, one of the two servers may adjust its clock so that the link TS of the received SERVER message becomes "now". It does so if it does not have RELIABLE_CLOCK set, if the received boot TS is older than the existing boot TS, or if the received boot TS is equal to the existing boot TS and the server initiated the connection.

Server Collision

If a server receives a SERVER message and the name or numeric of the new server already exists, there is a server name/numeric collision. This needs to be solved by breaking the link.

An implementation which will not hub (such as services) does not need to support this at all.

What to do when a server collision happens:

• If there is a name or numeric collision with the local server (myself) or with any U:lined (services) server, close the direct connection to the newly introduced server.

• If there is an existing server with the same name, but the numerics differ, or if there is an existing server with the same numeric but the names differ, remove the newly introduced server.

• If the newly introduced server is a direct connection and has an older or equal link TS than the existing link, close the new connection.

• If the newly introduced server is a direct connection, remove the existing server (ghost) and mark the direct connection of the new server as "caused ghost".

• If the direct connection of the newly introduced server is marked as "caused ghost", remove the existing server (ghost loop).

Ghost/ghost loop has not been clearly defined in previous specifications. When testing, a ghost loop (without a *real* ghost) could break every server link in the loop, ircu prevents this with another check which is not documented and which is somewhat impossible to understand in the source, and which is not documented in Carlo Wood's specification.

If none of the above criteria matched, break the second youngest link.

Why the second youngest link? Well, think of this situation: net split, 2 net fragments: A-B-C-D and E-F-G-H. The currently existing links are old/good links (such as between hubs). Two opers independently make a link. One tries to link A to H, the other tries D to E.

A-B-C-D
|     |
H-G-F-E

The new link/SERVER message A-H propagates from A to D and from H to E. The new link D-E propagates from D to A and from E to H. Somewhere in the middle, on servers B/C and G/F there will be a collision, but thats not a good place to break the link. Either the link A-H or D-E needs to be broken, and in both B/C and G/F the same decision must be made.

It is certainly possible to use more optimized ways to achieve the same result, but I believe this explanation leaves no doubt.

There is a "loop". The path between the 2 colliding servers. For example, servers ..A-B-C-D-A.. in this case, there's 4 links in the loop. This includes the newly introduced server which caused the loop. To get the second youngest link, sort the links by link TS from latest to earliest. Get the TS of the second link. From any/all links with that TS, get the link which has the greatest server name (sorted alphabetically). If multiple links have the same greatest server name, get the link which has the other server name be the greatest.

Example 1: loop: A-B/103 B-C/101 C-D/102 D-A/104. second youngest link is A-B.
Example 2: loop: A-B/101 B-C/101 C-D/101 D-A/101. second youngest link is C-D.

One breaks the link by SQUIT'ing the server on the remote end of the link.

Network Database Resynchronization

As discussed above, the connection is established by sending a PASS message, and a SERVER message. One may send PING messages, and one must reply with the proper PONG messages (in long/"non P10" format). One must not send anything else in this stage. One may ignore other messages on the receiving end, such as any status notices.

Once the connection (server link) has been established and verified, the next step is to synchronize various databases between the two servers. A net burst is sent to tell the other end of the link about the complete network state. The server which initiated the connection should wait until it has received the SERVER message from the accepting server, before it sends the net burst.

The messages that form the net burst must be generated in an order which allows the receiving server to create the objects and relationships represented by each message, at the moment the message is received. So, first server objects must be created and then the client objects connected to that server, and so on. Likewise, a server that is connected closer must be created first before a server behind it can be introduced, etc.

Because servers may be rejected as function of which servers are introduced, it makes most sense to first introduce all servers before introducing clients. This in order to avoid processing parts of a net burst, creating client objects, which need to be destructed a second later again.

Therefore, the net burst is constructed as follows:

Servers

Server details are transmitted via SERVER messages, similar to the initial introduction message, with the following format:

<source> S <server_name> <hop_count> <boot_time> <link_time> <protocol> <numeric/maxconn> 0 :<description>

The syntax of this message is nearly identical to the originally received server message, with the exception being that the message is prefixed (i.e., <source>) with a numeric to indicate which server sent this message (and also, therefore, which hub this new server is linked too). There is also a fixed "0" present before the <description> field, which is just a placeholder for future use, currently unused.

A SERVER message is sent for all servers on the network. Before a SERVER message for a given server is sent, a SERVER message for any server in front of that server must have already been sent. This can be implemented by first sending SERVER messages for all servers with hop count 1, then for all servers with hop count 2, and so on.

Glines

Any currently unexpired global Glines are transmitted via GLINE messages along with lastmod TS.

Jupes

Any currently unexpired JUPEs are transmitted via JUPE messages with the following format:

<source> JU * <+|-><server_name> <lifetime> <lastmod> :<reason>

For example:

1  2  3 4                  5         6         7
AZ JU * +juped.darenet.org 000003593 955419707 :Juped Server

Notes:

1. <source> - The numeric of the server sending this message.
2. The JUPE message token.
3. The target that should apply this JUPE (always "*" during bursts).
4. <server_name - The name of the server to JUPE, prefixed with a "+" if the JUPE is active, or with a "-" if its not.
5. <lifetime> - The remaining absolute lifetime of the JUPE, expressed in seconds.
6. <lastmod> - An epoc timestamp of the last time the JUPE was modified.
7. <reason> - The reason the JUPE was applied.

Clients

Information on all clients/users is transmitted via NICK messages, of the following format:

<source> N <nick> <hop_count> <timestamp> <username> <host> [+modes] <base64 IP> <numeric> :<gecos>

For example:

1  2 3       4 5         6    7            8     9      10    11
AF N Client1 1 947957573 User userhost.net +oiwg DAqAoB AFAAA :Generic Client.

Notes:

1. <source> - The numeric of the server sending this message (hence, owning this client).
2. The NICK message token.
3. <nick> - The nickname of this client, currently max 30 characters.
4. <hop_count> - The hop count of this client (i.e., how many servers away it is on).
5. <timestamp> - An epoch timestamp indicating when the user was created.
6. <username> - The "user" part of the user@host mask.
7. <host> - The "host" part of the user@host mask.
8. [OPTIONAL] <+modes> - User modes. If present, this is always "+<user modes for this client>. The special +r usermode is followed by the client's account name; see documentation for ACCOUNT.
9. <base64 IP> - The real IP address of this client, a base64 encoded 32bit int.
10. <numeric> - The client's numeric (numnick), in SSCCC format.
11. <gecos> - Free form user info line.

BURST Messages

Channel details and membership information is synchronized on one (or more) "BURST" messages for each channel that exists, formatted as follows:

[NUMERIC PREFIX] B [CHANNEL] [CREATION TIMESTAMP] <+MODES> <ARG1> <ARG2> [MEMBER LIST] <:%BANS>

For example:

1  2 3        4         5      6   7  8                                         9
AZ B #darenet 949217470 +tinkl key 56 AAAAA,AAAAB,AAAAC,ABAAA,ABAAB,ABAAC,ACAAA :%*!*@*.net

Notes:

1. The numeric of the server sending this message.
2. The "BURST" token.
3. The name of the channel to which this data belongs. Currently #Channel and +Channel names can be sent in a BURST message, &Channels are not because by definition they are local to the server.
4. The epoch timestamp indicating when the channel was created.
5. [Optional]: Channel Modes.
   1. The channel may have a number of modes set, a swell as relevant mode arguments in the following 2 parameters.
6. [Optional]: Channel Key, this parameter is present if the channel modes contain a "k" mode.
7. [Optional]: Channel Limit, this parameter is present if the channel modes contain a "l" mode.
8. A comma seperated list of client numerics, with the following specific formatting rules to indicate +o, +v and +ov channel members.
   1. Numerics can have the following symbols appended on them; ":ohv", ":oh", ":ov", ":hv", ":v", ":h" or ":o". These indicate that this numeric is either Opped (:o), Halfoped (:h) Voiced (:v) and so on. This state applies to the numeric it is attached too, and all subsequent numerics until another state is encountered. For Example: AAABA:ov, AAABB:o,AAABC,AAABD,AAABE:v,AAABZ. Here, AAABA is both opped, and voiced, AAABB, AAABC and AAABD are opped leaving AAABE and AAABZ voiced.
   2. The first numeric of the member list will always contain a state symbol.
9. A space seperated list of bans present in the channel. The start of the ban stream is indicated by a ":%", everything following the ":%" is the ban list.

For Example:
:%*!*@*.foobar.net another!ban@*.com *!*fred@a.host.co.uk

Would add the following bans to the channel:
*!*@*.foobar.net
another!ban@*.com
*!*fred@a.host.co.uk

NOTE: We BURST excepts in the same manner.

If the length of a BURST message exceeds the maximum lengh of a line (512 characters) then the remaining channel members/bans are sent in subsequent BURST lines. The subsequent burst lines are only used to add additional members to the channel, and if necessary, channel bans. There will be no "Mode" parameters present. A sample additional burst line would be:

AZ BURST #darenet 949217470 ACAAB:o,ACAAD :%*!*another@*.ban.com

Summary

The following table summarizes the sequence of events that occur when a server connects (links) to another server. S1 is our server, while S2 is a HUB on the target network.

S1: Sends Password.
S1: Sends initial SERVER message.

S2 Confirms S1 has the correct credentials, and if so, proceeds. If not, S1 is squit with a relevant reason.

S2: Sends Password.
S2: Sends initial SERVER message.

S1 Confirms S2 has the correct credentials, and if so, proceeds. If not, S2 is squit with a relevant reason.

The following occurs asynchronously, however it is shown separately below for simplicity.

S1: Sends all the servers it is aware of as a stream of SERVER messages.
S1: Sends all the clients it is aware of as a stream of NICK messages.
S1: Sends the database of channel states on the network, as a stream of BURST messages.
S1: Sends all the jupes it is aware of as a stream of JUPE messages.
S1: Sends a END_OF_BURST token (EB) to indicate it has finished sending.

S2: Sends all the servers it is aware of as a stream of SERVER messages.
S2: Sends all the clients it is aware of as a stream of NICK messages.
S2: Sends the database of channel states on the network, as a stream of BURST messages.
S2: Sends all the jupes it is aware of as a stream of JUPE messages.
S2: Sends a END_OF_BURST token (EB) to indicate it has finished sending.

S2: Sends an EOB_ACK token (EA) to indicate it has succesfully recieved the END_OF_BURST from S1
S1: Sends an EOB_ACK token (EA) to indicate it has succesfully recieved the END_OF_BURST from S2

Example Session:

[WRITE]: PASS :54321
[WRITE]: SERVER irc.darenet.org 1 947957852 947957852 J10 AB]]] :DareNET Client Server.
[WRITE]: AB N MrFoo 1 947957852 ~me myhost.foo.net +diksw DAqAoB ABAAA :Mr Foo (foo@bar.com).
[WRITE]: AB B #mychannel 946101324 ABAAA:o
[WRITE]: AB EB
[ READ]: PASS :54321
[ READ]: SERVER server1.darenet.org 1 947901540 947958150 J10 AFAD] :A Generic Server.
[ READ]: AF S server2.darenet.org 2 0 947957585 P10 AZAD] 0 :[192.168.10.3] A Generic Server.
[ READ]: AZ S server3.darenet.org 3 0 947957607 P10 AIAD] 0 :[192.168.10.5] A Generic Server.
[ READ]: AF N Client1 1 947957573 Ident userhost.net +oiwg DAqAoB AFAAA :Generic Client.
[ READ]: AZ N Client2 2 947957719 Ident userhost.net +iwg DAqAoB AZAAA :Generic Client.
[ READ]: AI N Client3 3 947957742 Ident userhost.net +iwg DAqAoB AIAAA :Generic Client.
[ READ]: AI N Client4 3 947958121 Ident userhost.net +iwg DAqAoB AIAAB :Generic Client.
[ READ]: AF B #foo 947957734 +tink akey AIAAB,AIAAA:v,AZAAA:o :%*!*another@*.ban.com *!*foo@bar.net
[ READ]: AF B #darenet 947957727 AIAAB,AZAAA:o
[ READ]: AF B #another 946101321 AFAAA
[ READ]: AF JU * +juped.darenet.org 3600 947958100 :Broken, please fix
[ READ]: AF EB
[WRITE]: AB EA
[ READ]: AF EA

Continuous Operation

This section provides details of the various messages that can be sent after successfully linking to the network, and synchronizing the channel/user database.

Channel State Operations

There are a number of messages that can modify the state of a channel, these are:

MODE

The MODE message can modify channel modes and bans, and also give or take operator/halfop//voice status from channel members.

[NUMERIC PREFIX] M [CHANNEL] (+|-)[MODESTRING] <MODESTRING PARAMETERS>

For example:

1     2 3        4        5
AZAAA M #darenet +stinlko 500 TestKey BAC

Notes:

1. The numeric of the user issuing this MODE command. It can be assumed this user is opped on the target channel.
2. The "MODE" token.
3. The target channel.
4. The "Mode string".
   1. This consists of up to 6 '+' or '-' (add or remove) prefixed channel modes. (If no '+' or '-' are specified, a '+' is assumed unless a '-' has been encountered previously in the mode string). For example, '+s+t+n-l-io' is a valid mode string, as is '+stnmov'.
5. The "Mode string Parameters".
   1. This is a matching list of parameters to the modes supplied in the "Modestring".
   2. Example: If the Modestring is "+stnlo", a typical parameter string would be "500 AZAAA". The first 3 modes, 's', 't' and 'n' do not require parameters, so non are present. The following two, 'l' and 'o' both require parameters, so they are 500 and AZAAA respectively (This sets the channel limit to 500 users, and ops the numeric AZAAA).

NOTE: The "MODE" message is also used to modify a client's user modes, not just channel modes.

OPMODE

The OPMODE message is identical in syntax to the MODE message, however it will only ever have an operator as the source. It is likely that the source of this mode will not have ops in the target channel, but it should succeed never the less.

JOIN

PART

KICK

TOPIC

CLEARMODE

[NUMERIC PREFIX] CM [CHANNEL] [MODES]

Client State Operations

NICK

AZAAA N Nick2 955423230

MODE

AZAAA M Nick2 :+odi

ACCOUNT

AX AC AZAAA oper

The ACCOUNT message provides a way for servers, such as the channel service server, to set the account name information that is associated with a client. Once set, it cannot be unset or changed, and will be propagated in NICK during net bursts using the special user mode +r followed by the account name.

Channel/Client Messaging

PRIVMSG

NOTICE

CNOTICE

CPRIVMSG

Programmers reference: Client/Server Structures

This section provides information on the standard Client/Server structures, for easy reference during development.