This week’s Dojo is focused on view processing. We’re methodically implementing paging, sorting, filtering, grouping, and summaries for views defined on Objects or Composites, with a powerful pipeline architecture.

With this architecture  some processing happens directly within the datastore, some within the application server, and some within the web browser. All 10 datatype families are being implemented, with mindful consideration to many possible performance issues.

This is taking us deep into some pretty interesting algorithms directly influenced by group algebra. In a nutshell, we’ve constructed an abelian group to model the filtering and grouping rules defined by complex datatype families such as Chronological, Locational, and Numerical. It’s all pretty esoteric, but it’s allowing us to implement the whole thing with just meta-data and Formula.js expressions, making it fully extensible.

Demos coming later this week.

Thank you Niels!

Take a look at the Airbnb user interface. How could one build this with STOIC?

Well, it’s going to be quite easy once we’re done refactoring our datatype families. First, notice the button bar at the top right with LIST, PHOTO, and MAP. This is nothing more than a list of perspectives. LIST and PHOTO can be implemented using the Tiles perspective with two different layouts, while MAP can be implemented using the Map perspective. Now, take a look at the left-side block from which you can filter properties. How could we generate a canonical filtering block with similar capabilities?

A new version of the View widget will add a perspective selector and a filtering block. Now, let’s see how we could actually implement the latter: when you look at Airbnb’s user interface, you see a list of filters than can be set to refine a property search (Room type, Price, etc.). Obviously, these filters can be implemented as filters defined on the fields of a view. But this begs an interesting question: what defines the user interface control for a given filter? For example, Room type is filtered with a set of check boxes, while Price is filtered with a neat double slider. How can we decide which control to use, just from a field’s datatype?

The answer can be found when looking at datatype families. Following a recent refactoring, our 60 datatypes are now grouped in 10 datatype families, using the newly-created Family object:

• Chronological
• Identificational
• Locational
• Logical
• Media
• Numerical
• Referential
• Spatial
• Temporal
• Textual

While we were working on grouping rules, we realized that the same rules could be used for filtering, and that they roughly matched the set of datatype families that we had defined earlier. As a result, we decided to combine them together and to define datatype families in more details, with a dedicated object. From there, we added some meta-data that would let users specify family-driven filtering and grouping rules from the new view editor.

Last but not least, we defined a set of 10 controls (using the Control object), one for each datatype family. These controls are not used to display the fields of a datatype though. Instead, they’re used to set the parameters of filtering rules, which is precisely what you’re doing from the filtering block shown on the Airbnb user interface.

Now that we have a proper meta-model for it, we can implement this new View widget, and the whole Airbnb-like user interface will come out of it auto-magically, for any object (or composite), with any view, through any perspective…

I can’t wait to see that one working…

Challenge: Can anyone find a synonym for media that ends with ‘al’?

As we’re building our new view editor, we’re finally taking the time necessary to specify some of the more advanced behaviors of views. One of them is grouping, which is a lot more complex than it might seem at first sight.

Grouping usually allows you to group the records of an object by category, for example orders grouped by products. Sometimes, groups apply to periods. For example, when you group by date, you need to specify a period like month or quarter. And when you group by a relationship, the possible groups are the records of the relationship’s target object.

Looking in details at our 60 datatypes, we figured that we could group them (no pun intended) into 12 main categories, for which we will implement slightly different grouping rules. Some are pretty simple, while others are more complex and will let users specify parameters at runtime, such as a period for a grouping by date.

Address
The user can select a grouping level:

• By route
• By locality
• By code (aka Postal Code)
• By county (aka Administrative Area Level 2)
• By state (aka Administrative Area Level 1)
• By country
• By quadrant (NE, NW, SE, SW)
• By North-South hemisphere
• By East-West hemisphere
• By latitude (with optional increment)
• By longitude (with optional increment)

Distinct
One group per distinct value of the field across all records.

Geocode
The user can select a grouping level:

• By quadrant (NE, NW, SE, SW)
• By North-South hemisphere
• By East-West hemisphere
• By latitude (with optional increment)
• By longitude (with optional increment)

List
This grouping rule is used for the Boolean, Tristate, and Variation datatypes, for which the list of groups is known in advance (Yes, No, and Maybe for Tristate for example).

Media
This grouping rule is used for all datatypes of the Media family (files).

The user can select a grouping level:

• By media type
• By media subtype
• By file size (with optional scale and increment)

Number
Most datatypes of the Numerical family use this grouping rule. Numerical values are converted to decimal values and compared to the minimum and maximum values of each group. The set of groups is defined by the minimum and maximum values of all field values for all records.

The user can select a grouping scale:

• Linear (with optional increment)
• Logarithmic (with optional increment)
• Percentile (with optional bracket)

Options
This grouping rule is used for the Category and Workflow datatype, for which the set of groups is the set categories or the set of steps respectively.

Percent
The user can specify a percentile bracket.

Period
The user can specify a grouping period:

• By second
• By minute
• By hour
• By day
• By week
• By month
• By quarter
• By semester
• By year

Reference
This grouping rule is used for datatypes which primitive is JSON and for which grouping is done against a particular value within a JSON object. The rule defines a reference to the value and its datatype. The actual grouping rule is the one defined for the referenced value’s datatype.

Relationship
This grouping rule is used for the Relationship and Advanced Relationship datatypes. In the latter case, only single relationships are supported. The possible groups are the records of the relationship’s target object. 

None
19 datatypes, most of which of the Textual family, do not support grouping.

Now that we have a better understanding of what grouping means, we can implement it. For this purpose, we need to do two things: generate a set of groups from a given parametrized grouping rule (a rule with all parameters instantiated), then create the filtering conditions that can be used by the database (through our object data mapper) to return the records of an object for any given group. This will require the implementation of two JavaScript functions for each grouping rule, getGroups and getGroupFilters respectively. I’ll try to write these two functions, then pass them on to François so that he can integrate them within our user interface.

Fun stuff!

When I woke up this morning, I found a couple of emails in my inbox notifying me that a couple of our stack.stoic.com registered users had suggested 5 more datatypes that could be added to our growing list (55 at last count). Here they are:

• Binary
• Hexadecimal
• IP
• Octal
• Range

Congratulations to Madhav and Stephen who got themselves cool badges.

We now have 50 datatypes defined for the platform, thanks to recent suggestions:

• Presentation by Michael Hay
• Sequence by Andréguy Dubar
• Timer by Dan Oneufer

All three will get a cool Datatype badge!

We just added three more datatypes to our collection:

Color

This will be used to define the color of user interface elements such as icons, using the CSS syntax. Its associated form control will be implemented using Colorpicker for Bootstrap, extended with a simple drop-down menu allowing the user to select a logical color (info, warning, danger, etc.).

Repository

This will be used to reference a GitHub repository or a Gist.

Tristate

Yes, No, Maybe.

I’m almost done with the Engineering functions of Formula.js, making my way through all the functions dealing with complex numbers. In order to take advantage of these functions, I’ve decided to add a Complex number datatype.

We just implemented the Image Datatype. Here is what the Object View looks like with it:

When clicking on an image from the Record View, here is what you get:

You can add any image stored on Google Drive:

And you can also add a new image to Google Drive:

Now that we have this infrastructure in place, we will add support for Movies and Soundtracks.

Properly managing complex datatypes has proven to be quite a challenge with previous iterations of our platform. Therefore, we decided to take advantage of our complete codebase overhaul to implement them the right way, right away.

First, we opted to implement syndicated objects such as Contacts or Events as first-class objects rather than datatypes. As a result, the list of complex datatypes capable of storing advanced data structures is limited to two: Option List and Relationship.

Second, we bit the bullet and implemented both the Option List and Relationship datatypes with the advanced features that would be quite difficult to add afterwards: support for multiple options in an Option List field, and support for dynamic target objects in a Relationship field. These two features added quite a bit of complexity to our code, but we’re done with them now, and we can take advantage of them for many other features that will essentially come for free.

All this plus the work on our database administration tools took us quite a bit off track, and no work was done on the record view, even though it was pretty high on our list of weekly goals. Also, we had to spend some time on the closing of a major deal, which further added to the distraction. Let’s hope that we can catch up tomorrow…

Over lunch, François and I devised a way to simplify our numerical datatypes. In a nutshell, the currency of a currency amount will be defined at the field level for all records of an object, not at the level of an individual currency amount within an individual record. The same goes for quantities that need a unit of measure (like a distance expressed in kilometers).

When you think about it some more, it actually makes a ton of sense, because supporting different currencies or units of measure within the same field would make it very difficult or even impossible to compute summaries (like average or sum). In fact, the more I think about it, the more I believe that record-level currencies and units make no sense at all…

From a user interface standpoint, we created a QuantityBox widget, which automatically displays a datatype-dependent quantity unit, such as a currency symbol, a unit symbol, or the percent sign. In order to deal with the fact that some units of measure might require multiple characters, we switched to a monospaced font (Courier New in this screenshot).

From a meta-model standpoint, we extended the meta-data of our datatypes with an extra dimension that we call quantity. In the case of the Percent datatype, it statically references the percent sign (%). But for Currency or Quantity, we reference options of the datatype, respectively Currency and Unit. This makes it fully extensible, while remaining quite simple.

Now, I just wish that our user interface framework would support nicer looking fonts…

No matter how simple you want to keep things, numerical datatypes are really complex to deal with.  Of course, you have the usual Integer and Float, for which you need to decide whether you keep them separate (more clarity), or merge them into a single Number datatype (less datatypes). Here, there is no perfect answer, but my personal preference goes toward keeping them separate.

Then, you have Percent, which is usually stored in the database as a float (numeric), but is manipulated by users through its value multiplied by 100 (80% instead of 0.8). In this case, you want the widget used to capture values of this datatype to show that it is dealing with percentages instead of floats. We achieve this goal by aligning the percentage value to the right and displaying a non-editable percent sign next to it in a slightly lighter shade of grey. Clean and simple.

But there are even more complex numerical datatypes: currency amounts (eg. $9.95) and measured quantities (eg. 5.25”). These need to store two pieces of information: a value and a unit (currency or unit of measure), which makes it harder to properly store them into a database. And things get really complex when you have to deal with our user interface framework, which caches all this information in idiosyncratic ways that vary depending on the datatype being used.

All this is making our model-driven representation of datatypes all the more complex, and I’ve reached a point where I’m not exactly sure where everything should go. I could create additional meta-data dimensions to model all this complexity, but this would reflect very poorly onto the code of our Data CRUD API and Record View. Or I could take a break and look for simplicity.

I think I’ll take a break…

And now, make way for our most ambitious datatype yet: Boolean.

This datatype can store one of two values: TRUE or FALSE. If the value is TRUE, the CheckBox is checked. If the value is FALSE, it is unchecked. If you check the CheckBox, the value is automatically set to TRUE, and if you uncheck it, it is set to FALSE.

Voilà !

What makes this datatype so special are you asking? Well, because it does nothing more than it is supposed to do. No images, no links, no support for three-state or fuzzy logic. None of this advanced stuff that we like so much. Just a simple CheckBox. And believe me: sticking to these basic requirements and not developing another case of overengineeringitis was quite a challenge, which makes this datatype our most ambitious yet.

Here are some more pixels, this one for the Address datatype. Nothing really fancy there. You just type an address, and a Google Maps is automatically added. The address itself is stored as a string, because simple is beautiful, and because that’s all you really need when you have Google around. The icon on the right opens the address in Google Maps, because we could not find a way to make the map’s image itself clickable (go figure).

We just finished the user interface for the Event datatype. Much like the Contact datatype, it is used to reference an externally-managed object, in this case events on a third-party calendaring application, through a pluggable API.

Once we fully implement its lookup function, you will be able to type an event’s name, an event’s id, or a date in the TextBox, then select a matching event from a ListBox. The event’s date links to the week where the event appears on the calendar, and the event’s location links to Google Maps. This is another example of how powerful virtual objects can be.

PS: François Beaufils wasn’t actually there, but I needed to add a guest for testing purposes…