I am Irvin Lim.

I make pretty and functional things on the web.

CS3216 Assignment 1

With only a few more days till the mid-assignment submission, I thought it might be a good idea to share and reflect on what we have achieved and I have learnt during the process thus far.

Choosing a framework

My team consists of Leon, Thanh and Eugene, and since we did not have much time for the assignment, we had to work in a framework and language where all of the developers are comfortable with. Our common language was JavaScript, and hence we decided to go for Node.js frameworks. Even though Leon and I had extensive experience working with Meteor.js (with React and MongoDB), we wanted to try out a different stack that perhaps would be less feature-filled than Meteor (which may add to code bloat and development time). We initially decided to go with react-starter-kit, but because none of us have any experience with many components of the boilerplate (especially Express), we found it difficult to even get the some code up and running.

I think the problem here lies in not being familiar with the standard practices in Express (e.g. req and res), and with the deadline looming up ahead, it was quite particularly stressful to have to learn a concept without having anyone to guide you along. Thus, we made a tough choice to revert back to Meteor.js only on Monday, with no lines of code at all yet. It went much smoother than expected, and since Leon and I are rather proficient in Meteor, we could guide Thanh to pick up the ideologies and various concepts of Meteor within only one night. I think that the lesson here would be that having to learn a framework without anyone to ask for help in person is not advisable in a time-sensitive situation like this. I found myself still learning even more despite using a framework I am familiar with.

Working with MySQL in Meteor

Meteor comes bundled with MongoDB as its database, with many of its parts tightly bound to MongoDB and the client-side library Minimongo. Having to switch to MySQL as our database meant that quite a number of parts would require us to rethink and simplify, or otherwise we would have to resort to hacky methods.

For example, the user accounts package for Meteor would easily implement user authentication using Facebook or other methods, but it was so tightly coupled with MongoDB that it was kind of impossible to make use of in this context. I decided to use Sequelize as our ORM to replace Meteor’s ORM-like functionality for MongoDB, which was one thing I needed to pick up quick. I found a package for Facebook Dialog Login called react-facebook-login which was good for in how easy it was to set up. Hence, I simply had to take the response from after the user clicks the Login button, and somehow turn that into a user session, yet maintaining a rather secure implementation of it.

It isn’t as simple as it sounds. I found that cookies are susceptible to Cross-Request Site Forgery (CSRF) and thus decided to go with using Meteor’s Session variables which are stored in localStorage, whereby other websites are not able to access. I also decided to use JSONWebTokens (JWT) since it was the new in thing, and it seems that it helps in terms of security as well, since tokens are encrypted with a secret key. The final step was to link the user token to an actual user object (Solution: make database requests on client startup), and pass it to Meteor so that the current user object can be accessed globally, yet still making it reactive (Solution: Session variables are actually reactive). There is still quite a while more to go before I can safely say that Meteor can be comfortable with MySQL. Prior to this, I wasn’t really optimistic about venturing into Meteor and MySQL since it strips us of the ability to use many Meteor features such as Minimongo, Pubs/Subs, and Optimistic UI. It was super unnerving to not have confidence in our approach, but I’m glad we went for it anyway, and successfully recreated a simple authentication mechanism.

Thus, I think that the lesson to be learnt here would be to JFDI, if it seems like the best decision, and not to worry too much about how efficient or stable it is. Only during the process can you more reliably assess whether your current approach is sensible or not.

Finding data and APIs

Our application aims to help students who are about to go for exchange to find useful information relating to their exchange, as well as to connect with other exchange students attending the same university as them. We needed to find quite a lot of data without having to key in stuff manually, such as:

  1. List of countries and their flags
  2. List of universities and their logos
  3. News API localised for the country
  4. Exchange rates API and so on…

The hardest data to source for turned out to be the listing of universities in the world. With no formal API, we were at a loss of what to do, and we were prepared to write a scraper to get data from certain listings. I stumbled upon topuniversities.com and found that they have listing pages which have pagination that didn’t require the browser to reload the page, and so I was hoping to be able to find an AJAX request that could possibly expose some sort of API for us to use. I ran into luck at the QS University Ranking page - lo and behold - the page is retrieving all of its university data (918 universities, including name, country and logo) from a .txt file!!!

I was so excited to have found it and I immediately went ahead to parse the data so that it could be consumed by our database. This was really exciting as I was not expecting to be able to find data in such a manner.