Waartaa 0.2 is here! It's responsive, sleek and faster. Once again, a big thanks to the contributors to help us with Waartaa. Following is a summary of changes released in waartaa 0.2:

  • Improved client side routes
  • Mobile optimized UI
  • Server side tracking of unread logs count and unread mentions count
  • Lazy loading of chat logs, only when needed
  • Cache subscriptions across routes using subs-manager
  • Infinite bidirectional scrolling of chat logs using client side routes, maintaining only a window on N (configurable) pages of logs at a time
  • Permalink for each chat log
  • Update to Meteor 0.9.3
  • Improved Ansible scripts for setting up servers and deploying Waartaa

Source code: https://github.com/waartaa/waartaa/tree/0.2

Comments


After 0.1 release of Waartaa, we've started our work to make Waartaa usable in mobile devices. Here's a sneak peek at our initial mockups for the mobile UI. Please share your valuable feedback on this to help us proceed in the right direction and build a better communication tool for the community.

Comments


It's a great pleasure to inform you that we have released Waartaa 0.1, finally. Again, a big thanks goes to the team of contributors for Waartaa to make this happen.

This release includes all the basic IRC functionalities, centralized logging, notifications, etc. implemented in release v0.1-rc1. We worked on a lot of optimizations and automated the deployment process for Waartaa to make using and maintaining Waartaa as simple as possible. You can get the source in the following way:

1 $ git clone https://github.com/waartaa/waartaa.git
2 $ cd waartaa
3 $ git checkout 0.1

What's new?

  • Backend optimizations
    • OPLOG support
    • Rate limit tasks using queues
    • Async calls to database
    • Improve MongoDB performance in limited RAM
    • Reduce storage complexity for channel logs
  • Client optimizations and improvements
    • Smooth rendering of chat logs
    • Use latency compensation to make improve user experience
    • Optimize client side memory usage
  • Upgrade to Meteor 0.8.1
  • Easy server provisioning
  • Easy app deployment
  • Vagrant script for easy setup of dev sandbox

Anatomy

/galleries/waartaa/waartaa_0.1_stack.png

Currently, Waartaa runs as a monolithic app. In the future, we'd be dividing Waartaa into several components/layers so that it becomes possible to horizantally scale a Waartaa service.

Comments


Waartaa, a web based IRC client as a service, started out as a tool for people to stay online on IRC 24x7 along with centralized logging, notifications on mentions and unique identity across multiple devices. Waartaa applied to GSoC this year under The Fedoraproject.

As soon as we proposed the idea, it received an overwhelming response from the aspiring GSoC'ers. Waartaa received quite a few applications and the mentors had a hard time selecting the best proposal out of it. The ideas proposed were:

  • Build a central hub for searching/reading channel logs for Open Source communities and projects.
  • Build a faster and scalable backend.
  • Freedom of choice: Expose an API so that users can use their existing IRC clients with waartaa.
  • Find a secure way to authenticate with IRC services without storing RAW passwords.
  • Respect user privacy: user personal messages should be stored in an encrypted format in the server.
  • Allow users to download chat logs in various formats compatible with popular IRC clients.
  • HTML5 mobile app
  • VCS, Bugzilla and other task management tools integration.
  • Video/audio conference facility on top of HTML5 and JS technologies.

Lalit Khattar, an undergraduate student of Metallurgical Department of IIT Roorkee, India got selected as the GSoC 2014 candidate for Waartaa.

/galleries/waartaa/lalit_gsoc_selection_tweet.png

He will be building central hub for searching/reading channel logs for Open Source communities and projects, basic integration of Video/Audio conference facility using WebRTC and admin pages for Waartaa.

What could be a better way to spend the community bonding period than starting to contribute code to the project even before the actual GSoC starts? Lalit started to work on writing tests for Waartaa and this'd give him a better idea of the codebase. He will be going to start with implementing his proposed project idea as soon as the GSoC Coding Period starts. The next three months is going to be an awesome rollercoaster ride for the mentors and the candidate as well.

Happy hacking!

Comments


Overview

Waartaa is a realtime web IRC client as a service. Waartaa is powered by:

  • Meteor JS
  • MongoDB
  • node-irc, amongst other cool NPM packages
  • a bunch of meteorite packages from atmospherejs.com

We'd released version 0.1-rc1 a few months back with some basic feature set.

  • Send server, channel, personal messages
  • Support basic IRC commands: join, away, me, whois, msg, part
  • Centralized logging for every user for every channel, server
  • Realtime update across multiple clients
  • Mail notifications on nick mention when the user is away

Issues

  • High storage complexity

    Storing logs per channel per user led to high storage complexity.

  • High CPU usage

    Meteor 0.6 was not optimized to handle heavy writes to a collection which was published in realtime. In our case, the UserChannelLogs collection was the one published in realtime and with heavy writes. Meteor would use to poll the collection to find and push updates on the existing publications. The increased reads led to increased CPU usage and network IO for the MongoDB service. CPU usage would spike to 400%. Also, handling too many user tasks in the application server led to CPU spikes.

  • Slow app performance

    By default, Meteor apps work synchronously, unlinke other typical nodejs apps. However, Meteor being a NodeJS framework itself, it's single threaded and it'd make other requests wait until it's done with processing the current request, which mostly involved waiting of database responses.

  • High RAM usage

    The MongoDB server for https://try.waartaa.com runs on a mere 1 GB memory. With the new users listening to variegated channels, the size of the collection spiked in a few days. Because, the same collection was also the most heavily read collection, this led to increased RAM usage and finally RAM starvation, We started experiencing latencies in write to MongoDB and the realtime app was no longer realtime.

  • Difficult setup

  • Painful to deploy

Working in limited resource sometimes has its benefits. We were forced to optimize Waartaa to make it usable in the real world.

Optimizations & solutions

  • The OPLOG hack

    The meteor community came up with a hack using MongoDB OPLOG to make MongoDB a really live database. Initially it was SmartCollections, then the OPLOG driver in Meteor 0.7 and finally an improved OPLOG driver supporting complex MongoDB queries in Meteor 0.8. OPLOG is a proven technology used by MongoDB to replicate databases. Meteor is now able to check for publication updates just by following MongoDB's OPLOG stream. So, goodbye to polling and high useless reads. With the OPLOG driver, waartaa's MongoDB instance finally became stable and the CPU usage came down to under 5% on an average.

  • Async read/writes to DB

    Meteor comes with an option to do DB read/writes in an asynchronous way. Switching to async DB read and write wherever possible made waartaa more responsive by leaving room for the app to handle new requests while the app is waiting for DB IO.

  • Log once per channel

    Logging once per channel was a challenge for Waartaa as we don't have a single bot to listen to. We worked on a crude algorithm to do this. We pick up a few clients from the pool of available clients listening to a channel that waartaa will monitor for new channel messages. We use CappedArray to keep a track of a limited number of recent channel logs written to the DB and use it to determine if a new channel log from a client should be written or not. Now, we save channel logs in ChannelLogs collection instead of UserChannelLogs.

    This has helped Waartaa reduce the storage complexity for channel logs to O(c), where c is the number of channels Waartaa is listening to.

  • Get MongoDB perform better in less RAM

    Waartaa now stores only a limited number of recent logs per channel in the ChannelLogs collection to keep the size of this collection in check and simultaneously archives older logs. This reduces the size of the working set of data needed for realtime publication and hence, it reduces the memory usage of the MongoDB service.

  • Task queues

    Some operations like joining servers and channels during app restart were very costly operations and would lead to high CPU usage for some time. Hence, the application would become slow during server restart. Similarly, there were instances when there'd be high number of requests for updating data for channel nicks, nicks leaving/joining channels, etc. This would lead to CPU spikes. Implementing in memory task queues helped to rate limit such operations and keep the CPU load in check. In the longer run, we need to consider better task queuing solutions.

  • Easy sandbox setup with Vagrant

  • Automated deployment with Ansible

Below is a snapshot showing the CPU, network load on the app server of https://try.waartaa.com during the initial days of testing our optimizations.

/galleries/waartaa/waartaa_app_optimize_cpu_usage.png

The above steps have enabled us to keep our demo instance running at https://try.waartaa.com using limited resources. The current model of Waartaa works fine as a single standalone instance. It's not that great for horizontal scaling yet. It's in our roadmap to make waartaa scalable.

For now, this is all we have to offer for our upcoming 0.1 release.

Comments


https://1-ps.googleusercontent.com/sx/s.google-melange.appspot.com/www.google-melange.com/soc/content/2-1-20140305/images/gsoc/logo/920x156xbanner-gsoc2014.png.pagespeed.ic.gdr4t3Igca.png

We are elated to inform you that Waartaa has been accepted as a project idea for GSOC under Fedora's umbrella.

For those who are new to Waartaa, Waartaa is an open source communication tool for teams and groups. It is built on top of IRC. Currently, Waartaa is an IRC client as a service and it supports centralized logging, 24x7 idling, notifications and unique identity to a user on IRC. The source code is available at https://github.com/waartaa/waartaa for you to download and hack on it.

If you find Waartaa interesting and something that you need, please feel free to ping us at #waartaa on Freenode. If you are a student, you are most welcome to submit your project idea for GSOC 2014 to help make Waartaa awesome.

Mission

Our mission is to implement a one-stop open source communication and collaboration tool for teams and groups, built on top of open source technologies (IRC, WebRTC, etc.).

Current status

Waartaa is no longer just an idea in pages, it's real, it's live. You can try our demo instance at https://try.waartaa.com. We have also started to receive quite some contributions as well. Lately, Waartaa has seen some massive improvements in terms of optimization and scaling. You can download and run Waartaa in your server or machine very easily. This allows you to get your own IRC client as a service up and running in no time.

Technology stack

Roadmap

  • Build a central hub for searching/reading channel logs for Open Source communities and projects.
  • Build a faster and scalable backend.
  • Freedom of choice: Expose an API so that users can use their existing IRC clients with waartaa.
  • Find a secure way to authenticate with IRC services without storing RAW passwords.
  • Respect user privacy: user personal messages should be stored in an encrypted format in the server.
  • Allow users to download chat logs in various formats compatible with popular IRC clients.
  • HTML5 mobile app
  • VCS, Bugzilla and other task management tools integration.
  • Video/audio conference facility on top of HTML5 and JS technologies.

Communicate

Comments


It's with great pleasure that we are releasing Waartaa v0.1-rc1. It'd never have been possible without the support of our contributors, users and well wishers. As of now, Waartaa packs in:

  • Basic IRC functionalities:
    • Join server and channels
    • Send PMs
    • Registering and identifying nicks
    • Change nick name
    • Change status
    • WHOIS
  • A few RAW commands are also supported:
    • WHOIS
    • JOIN
    • AWAY
    • PART
    • me
    • msg
  • 24x7 idling
  • Mail notifications when away
  • Central logging
  • Uniform access across multiple environments.

We have now frozen development of new features for Waartaa and will be focussing on bug fixes, fine tuninig and optimizations till the release of v0.1.

Please give waartaa a try at https://www.waartaa.com and share your valuable feedback at https://github.com/waartaa/waartaa/issues?state=open.

Comments