Month: September 2019

Rebuilding the notifications flow

September 19, 2019 / Thomas Mertz / Comments Off

It’ll probably be about 2AM until this post goes out, so if I am some sort babbling incoherent dingus, I apologise in advance. Have a kitty.

OK. So, notifications. What an absolute clusterfuck.

I’ll see if my sleepdeprived brain can somehow map out the idea and model we have for notifications in Kin. The challenge is to be able to logically group and display the notifications, so we don’t give people a heartattack when they log in and see the list of activity while they’ve been away.

The achieve that I am using a table called ‘secretprefix_notifications‘. Clever, right? It looks a little like this:

CREATE TABLE `notifications` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `contentID` int(11) DEFAULT NULL,
  `contentType` varchar(10) DEFAULT NULL,
  `senderID` int(11) DEFAULT NULL,
  `senderType` varchar(10) DEFAULT NULL,
  `recipientID` int(11) DEFAULT NULL,
  `recipientType` varchar(10) DEFAULT NULL,
  `url` longtext,
  `notificationType` varchar(100) DEFAULT NULL,
  `timestamp` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8mb4;

The two content tables can tell us which specific piece of content we’re targetting. The sender ID does the same and (along with the same model for recipients) allows us to target both Users and Pages. Finally the URL will allow me to create a redirect link, and so somewhat invisibly notifiy the system that the user has not only seen this notification, but also reacted to it.

Finally, we have the column, notificationType. This column dictates, essentially, which text string to output when listing the notifications. Stuff like “Frank, Janice and 4 others have reacted to your comment.”

But … who gets notifications?

That’s a good question, that no sane person would ask, because reasons.

Depending on the event that triggers the notification it might be as banal is just dumping in the notification directly between two users. We would do that for strictly one-to-one events, like anything friend request related (request, acceptance or rejection).

But for stuff like status updates there’s a real chance that I might have to send notifications to a ton of people.

To do that we have a subscription model, not unlike the Notification Subscription function Facebook has (in fact, Kin 1 had that functionality before Facebook … so suck it, Zuck). Whenever a user interacts with an update, in any way (which is essentially reacting to an update, comment on an update, or reacting to a comment on an update) we create a subscription for new activity on that piece of content for that specific user or page.

That is stored in an equally originally named table called ‘secretprefix_subscriptions‘ and it looks like this:

CREATE TABLE `subscriptions` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `postID` int(11) NOT NULL,
  `subscriberID` int(11) NOT NULL,
  `subscriberType` varchar(10) DEFAULT NULL COMMENT 'Can be either ''user'' or ''page''',
  `active` binary(1) DEFAULT '1',
  PRIMARY KEY (`id`),
  KEY `kin_subscriptions_idx` (`postID`,`subscriberID`,`active`)
) ENGINE=InnoDB AUTO_INCREMENT=20 DEFAULT CHARSET=utf8;

We make a query to that table, excluding the author of the current activity, and pass that to the Notification class, that then creates a notification for each subscriber. The logic itself is pretty straightforward, once broken down.

Next step … notification output and rendering. But that’s for another post. Now I need sleep.

When Notifications go awry

September 15, 2019 / Thomas Mertz / Comments Off

So, I am looking into implementing Notifications for Kin 2, and while I want to make certain improvements over Kin 1, it’s important that we also don’t make the same mistakes again.

In Kin 1 we had one very strange bug. Notifications send to a specific User might end up in the Notification Queue of a given Page, and vice versa. It’s been an ongoing issue for a long long time, and both I and other members of the Kin 1 team struggled to understand why that was happening.

Well, I think I figured it out.

But to properly explain I’m going to have to back up a bit.

The very first iteration of Kin lacked a lot of the more advanced stuff we have today. There were no Pages or Groups. We still wanted Notifications for when people interacted with Updates, so Daniel built a subscription model. Basically every single time someone interacted with an Update, a row would be inserted into the subscription table denoting whether or not they had subscribed to that piece of content. The table looked a little something like this:

CREATE TABLE `subscriptions` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `updateID` int(11) NOT NULL,
  `userID` int(11) NOT NULL,
  `active` binary(1) DEFAULT '1',
  PRIMARY KEY (`id`),
  KEY `subscriptions_idx` (`updateID`,`userID`,`active`)
) ENGINE=InnoDB AUTO_INCREMENT=5067 DEFAULT CHARSET=utf8;

The two columns ‘updateID‘ and ‘userID‘ connects this table to the Updates table and the User table. The ‘active‘ column denotes whether or not the subscription is active.

This model had a lot of benefits. For instance, a User had the option of subscribing to an Update without interacting with it, essentially allowing people to lurk in a conversation and still be notified when something new happened.

What would happen is that whenever an interaction took place we would look at the Subscription table to determine who should receive a Notification. By polling on the ‘updateID‘ and ‘active‘ we could get a list of userID’s that were subscribed to notifications on this particular Update.

However, when the application grew, and specifically when we added Pages we added another set of parameters under which an Update could be posted. Essentially, once logged in you could “switch” to any of your characters Pages and then post as that page, with Updates having the pages logo and name in the Update, rather than that of the character.

What we, now obviously, forgot, was to add this functionality to the subscription model. All our tables have Primary Keys which we use to internally reference a bunch of data. But for this, essentially if a Page had Primary Key 11 in the Pages table and a User had Primary Key 11 in the Users table, those could get mixed up in the Subscriptions table, and presto there’s the crossed wires mess.

Now I present our new and improved Subscriptions table which hopefully (knock on wood) fixes this particular issue.

CREATE TABLE `subscriptions` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `postID` int(11) NOT NULL,
  `subscriberID` int(11) NOT NULL,
  `subscriberType` varchar(10) DEFAULT NULL COMMENT 'Can be either ''user'' or ''page''',
  `active` binary(1) DEFAULT '1',
  PRIMARY KEY (`id`),
  KEY `subscriptions_idx` (`postID`,`subscriberID`,`subscriberType`,`active`)
) ENGINE=InnoDB AUTO_INCREMENT=5067 DEFAULT CHARSET=utf8;

Now … onwards and upwards to Notifications. Which is probably going to make me hate life on a whole new level.

Rebuilding the timelines

September 13, 2019 / Thomas Mertz / Comments Off

UPDATE: A friend (Hi Tommy) wise pointed out that the UNION and double selects could be handled by adding an OR clause. So… that’s what I did.

In the old Kin the timeline, ie. the basic view that included all the posts viewable by a given character was built in a less than great way.

Because we allowed for heavy segmentation; ie. there were essentially three kinds of timelines one could post on (Users, Pages and Groups) and there were several types of posts (Default, Relationships, etc) we needed to develop a system where a post was decidedly distinct from the timeline on which it occurred.

We’d call that with an timelineType IN (‘user’,’group’) type definition. That returned the ID’s of the individual posts we needed to pull. Because I was still getting my head wrapped around PHP’s OOP model at the time I ended up with something that was essentially a mess of looping through the rows of the timeline query, then doing separate calls for each row into the Posts table, and then when rendering those I did separate calls yet again to include Author data (because a Post Author could be both a User and a Page).

For Kin 2, I’ve managed to whittle all that down to a single SQL statement. Which just goes to show how much I’ve learnt since building the first version. We still have all the same data available and we can still page through it with OFFSET and LIMIT.

So here’s the new and improved.

SELECT 
	posts.id as postID,
	combined_timeline.timelineID, 
	combined_timeline.timelinetype,
	posts.authorID,
	posts.authorType,
    IF(combined_timeline.timelinetype = 'page', pages.name, characters.name) as authorName,
	IF(combined_timeline.timelinetype = 'page', pages.slug, characters.slug) as authorSlug,
	postType,
	posts.contents,
	timestamp
FROM (
	SELECT DISTINCT timelines.contentID, timelines.timelineID, timelines.timelineType FROM timelines WHERE timelineType IN('user','page') 
	UNION ALL 
	SELECT timelines.contentID, timelines.timelineID, timelines.timelineType FROM timelines WHERE timelineType = 'group' AND timelineID IN (7,8,10,11)
) as combined_timeline
LEFT JOIN posts ON posts.id = combined_timeline.contentID 
LEFT JOIN characters ON characters.id = combined_timeline.timelineID AND combined_timeline.timelineType = 'user' 
LEFT JOIN pages ON pages.id = combined_timeline.timelineID AND combined_timeline.timelineType = 'page' 
ORDER BY posts.timestamp DESC;

In the above snippet I’ve intentionally left out the OFFSET and LIMIT functionality. I’ll probably add some hard limits to that in the API.

Much improved version

And here’s the improved version, per Tommy’s comments.

SELECT 
	posts.id as postID,
	combined_timeline.timelineID, 
	combined_timeline.timelinetype,
	posts.authorID,
	posts.authorType,
	IF(combined_timeline.timelinetype = 'page', pages.name, characters.name) as authorName,
	IF(combined_timeline.timelinetype = 'page', pages.slug, characters.slug) as authorSlug,
	postType,
	posts.contents,
	timestamp
FROM (
	SELECT DISTINCT timelines.contentID, timelines.timelineID, timelines.timelineType FROM timelines WHERE timelineType IN('user','page') OR timelineType = 'group' AND timelineID IN (7,8,10,11)
) as combined_timeline
LEFT JOIN posts ON posts.id = combined_timeline.contentID 
LEFT JOIN characters ON characters.id = combined_timeline.timelineID AND combined_timeline.timelineType IN ('user','group') 
LEFT JOIN pages ON pages.id = combined_timeline.timelineID AND combined_timeline.timelineType = 'page' 
ORDER BY posts.timestamp DESC LIMIT 25

Rebooting Kin

September 10, 2019 / Thomas Mertz / Comments Off

Kin is dead. Long live Kin.

The old application had accrued too much technical debt, and it just wasn’t fun to maintain anymore. So I’ve killed it.

Please welcome Kin 2, as the worthy and shiny successor. Kin 2 will, as opposed to it’s predecessor, operate as a SaaS type product. It’s currently divided into three core products:

The API
The account management site
The client application

The API

The API is the core of Kin 2. It contains all the logic and database connectivity. Ideally, once it’s mature anyone and everyone should be able to build clients that access Kins API. Hopefully this will lead to many exciting use cases that I haven’t thought of myself.

myKin

myKin is the account management site. It’s meant as a single gateway for players and organisers to track their games and characters. Whether or not it’s a good idea only time will tell.

The Client

The client is essentially what used to be the UI of Kin. It’ll have friend lists, status updates, reactions and comments, and so much more.

There’s a whole lot of reinventing the wheel going on, but that also allows for refactoring and improvements.

But why?

I’m doing all this for a number of reasons.

It just wasn’t fun anymore. All the time I had to spent on the project was spent maintaining code and putting out fires. There was no time to improve upon the application.
At the early onset of the project some choices were made that allowed for it to get out the door quickly, but also shut off some avenues for the project. Primarily that of disconnecting the UI from the logic.
People have been asking for a better mobile experience for years. The best I could offer was an OK mobile site, but given the complexity that wasn’t as good as I wanted it to be. By rewriting the application to have an API core we open the opportunity of possibly at some point creating native mobile apps for the platform.
I had learned all I could with the current platform. There was no challenge left, and it was very bread and butter.