Author Topic: Finally.. we have the gossip protocol  (Read 2182 times)

Dirk Diggler on: August 27, 2015, 11:38:43 PM

  • FIMK Staff
  • *****
  • Posts: 486
    • View Profile
    • Krypto Fin ry
http://fimkchat.com/2015/08/experimental-lompsa-0-4-7/

-----------------------

Finally I got around to implementing a protocol I've had in mind for a long time.
The idea is extremely simple but at the same time extremely powerful.

With the gossip protocol everyone running a FIMK server can send a message to one or more of his connected peers and be pretty sure that same message is distributed as fast as possible among all other peers.

Nothing is stored however!
All that remains on any of the servers is the ID of the message. And only for a limited amount of time just to make sure we don't keep sending the same message around.

To actually do something with gossip messages the receiving party should be actively listening for them.

A party does not need to listen for all gossip messages!

What you do is create a special key which is built up from a sender, recipient and or topic (or any combination of that) and you'll subscribe to that key through two-way websocket client/server connections.

Now when anyone sends a gossip that matches that key, as soon as that message reaches the (p2p) server your client is connected to you will be notified.

Having a free distributed messaging protocol could potentially come at a cost.
That's why all transactions and blocks on the network always go first.

It is a very early stage and these extra measures have not been enabled yet but this protocol comes with excellent spam prevention methods.
Since every gossip message has a sender and every peer in the network has knowledge of that sender's balance we could easily require a sender to have a minimum balance, we could also limit the number of messages per minute/hour/day etc per account ..

Every gossip message is signed with a secret phrase which proves who send the message (same mechanism as signing blocks).
Every gossip message is timestamped and the timestamp is included in the signature, messages older than - or from more than 15 seconds in the future are discarded.

The live chat feature is just one of it's uses and is what is available in lompsa (used to be mofowallet) 0.4.7 now.

Lompsa 0.4.7 shows how with this relatively simple protocol we can create a sophisticated client side (javascript) distributed real-time communications platform.

# client side protocol implementation
https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/services/gossip-service.js

# messages are stored and served from indexeddb
https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/providers/gossip-chat-messages-provider.js
https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/providers/gossip-chat-list-provider.js

Lots of other uses exist. Free fast real-time communications between FIMK users is just one of them.
Eventually the protocol could be used for anything.. communications, games, live multi-sig, live trading etc..
FIMK Developer | GPG fingerprint: CEF2 7C39 43BE 6800 504E  71BC 7E87 A7B0 AC34 E2D5 | mofowallet.com | blog

warmach #1 on: August 28, 2015, 03:41:30 AM

  • Senior Fimker
  • ***
  • Posts: 225
    • View Profile
Has this feature been implemented in the api?  I could see this being a useful tool for communication with my forging pool.
Warmach's Forging Pool
http://fim.warmach.info/pool/
May the forge be with you...

Dirk Diggler #2 on: August 28, 2015, 10:41:17 AM

  • FIMK Staff
  • *****
  • Posts: 486
    • View Profile
    • Krypto Fin ry
I could see this being a useful tool for communication with my forging pool.

There is the SendGossip function in the API. It's not up on github yet so i've pasted a copy here (http://paste.ubuntu.com/12212780/ java).
The API allows you to send a gossip message to the network, it takes these parameters..

- id (String ID that you must calculate)
- message (Hex String max 1000 bytes)
- senderPublicKey (your public key in hex)
- topic (String numeric unsigned long - similar to numeric asset id or account id)
- timestamp (integer timestamp in NXT/FIMK epoch format)
- signature (Hex String - signature over message, topic and timestamp signed by you)

Working javascript implementation of it all is here: https://github.com/fimkrypto/mofowallet/blob/master/app/scripts/services/gossip-service.js#L274
That impl includes 256 bit AES encryption of the message contents (as can be seen here).

I could see this being a useful tool for communication with my forging pool.

Pool as in people or pool as in the server?

FIMK Developer | GPG fingerprint: CEF2 7C39 43BE 6800 504E  71BC 7E87 A7B0 AC34 E2D5 | mofowallet.com | blog

warmach #3 on: September 01, 2015, 01:48:35 PM

  • Senior Fimker
  • ***
  • Posts: 225
    • View Profile


Pool as in people or pool as in the server?

I guess I could do both.  I had originally planned for just people.  I wanted a way to send messages that did not need to live on the blockchain forever.

Is there a way that I can use a perl script to look for new messages from a certain point in time?  How else can I read incoming messages?
Warmach's Forging Pool
http://fim.warmach.info/pool/
May the forge be with you...

Dirk Diggler #4 on: September 01, 2015, 04:51:55 PM

  • FIMK Staff
  • *****
  • Posts: 486
    • View Profile
    • Krypto Fin ry
The gossip protocol comes without a built-in storage mechanism.
Gossip messages send to the network are distributed among all peers on the network and that's that, messages are not stored by any of the peers.

The gossip messages are in many ways similar to NXT/FIMK transactions.

- they can use strong encryption
- they always have a sender
- they can have a recipient
- they always have a timestamp
- they always have a topic (which is a free-form number similar to how there are transaction types)
- they all contain a powerful cryptographic signature which can be used to confirm the sender is who he/she says he/she is (this will almost certainly be used at a later stage to manage network access and/or priority for gossip messages - basically spam control)

So how does that work if I cant query the network for messages?

Well thats where the efficiency comes from, it's an event based system.
For anyone familiar with NodeJS or Javascript in general knows that event based systems can be made to scale since they run so efficiently.

For this to work each server (peer) in the network (at least the ones that want to serve clients) needs to be able to have clients register with them as listeners (done over websockets) and when that peer detects a valid gossip message it should forward that to it's registered client listeners.
But not to every listening client of course, only to those interested in that specific topic, sender or recipient (or combination of those).

If done from Perl it might require some reverse engineering the current javascript code but if you do that what you'd end up with is a real-time always on connection from your Perl script basically straight into the gossip network. You can basically code the Perl script to be a lot like a bot that auto-responds to messages it receives from the network.

Other people can have a real-time connection with your script without knowing it's IP address, that is since all messages are directed to any one of the peers in the gossip network. Because of this one-to-one-to-all setup between clients and servers and servers among each other the scale of the number of direct connections becomes enormous. And because each gossip message comes with a sender ID which is your FIMK address (until i add this to NXT-PLUS) each relay node (peer) in the network has access to the same strong identity and balance info as all the others, meaning it can stop spam data at the gate (keeping things really fast).

What can be built on such a free decentralized real-time message bus (since thats what it is basically) I dont know. But I know it's a lot, real-time direct chat is just one it's uses.

----------------------------------------------

So how to connect from Perl? Don't know about that, but i can point in the right direction as to where the important javascript stuff is.
If you look at the gossip-service.js you'll find the full client-side implementation of it all:

https://github.com/fimkrypto/mofowallet/blob/e5a7db321ff990bb44efe65d3c0a9a620c313f36/app/scripts/services/gossip-service.js

Not all of that is required of course to get a working prototype. I'm planning still on making an easy to use plugin system to tap into Lompsa, it will contain the most easiest gossip support possible I guess.

----------------------------------------------

So in short - what to remember is - Gossip is a free decentralized messaging protocol - making it possible for people around the world to have real-time, free, encrypted communications with each other.
FIMK Developer | GPG fingerprint: CEF2 7C39 43BE 6800 504E  71BC 7E87 A7B0 AC34 E2D5 | mofowallet.com | blog

Eliphaz Fimk #5 on: September 01, 2015, 11:10:53 PM

  • FIMKrypto coordinator
  • Administrator
  • *****
  • Posts: 806
    • View Profile
    • FIMKrypto
Quote
Other people can have a real-time connection with your script without knowing it's IP address

Some exceptional long term work from Dirk, the materialization of which we're seeing now in the past few days. real-time, free, encrypted, anonymous communications application. Think about that!  :cheers:

kani #6 on: September 02, 2015, 01:48:27 AM

  • Senior Fimker
  • ***
  • Posts: 154
    • View Profile
Some exceptional long term work from Dirk, the materialization of which we're seeing now in the past few days. real-time, free, encrypted, anonymous communications application. Think about that!  :cheers:

Outstanding!  FIMK development continues to impress.  Keep up the great work!   :thumbup:
BitShares - Your share in the Decentralized Exchange