I Got 99 Problems but a Siren Ain’t One
Introduction
You know what’s easy? Building JSON responses for your web services. Write a DTO, maybe you need to throw a few annotations on to express how it’s serialised. Bish bash bosh. Job done! Ooh, this one is a little more complicated. I need to specify a custom serialiser. No problem.
You know what’s more difficult? Designing good and useful APIs for your web services. Ones that express more than mere properties and values. Ones that express relationships. Ones that express actions. Ones that express context and intent.
Expressing context and intent in API design is a noble goal. Providing information about what users can expect from your service is really useful. It goes beyond providing data, but also helps the developer consuming the service understand why the data is important, how different data entities are related, and what actions can be done with it.
My investigation into API design stemmed from a need to offer customisable role-based access control in a personal project. The service calculates access levels to resources based on the roles in the user’s access token. It also provides the actions that can be performed on a resource in its response. I was exploring the most effective way to convey this through the API.
This article explores ways you can add context and intent to your API, and describes how I go about doing so in my own development. I will touch on what’s available and what I thought was best. As a bonus, I talk about errors. Because I don’t enjoy designing those, either, and they are just as important a part of the API as the data you send.
HATEOAS
Hypermedia As The Engine Of Application State, also known by the truly horrible acronym HATEOAS, is probably the first option you will come across when investigating adding context and intent to your API definitions. As as example of how to use it, let’s steal the example from Wikipedia:
{
"account": {
"account_number": 12345,
"balance": {
"currency": "usd",
"value": 100.00
},
"links": {
"deposits": "/accounts/12345/deposits",
"withdrawals": "/accounts/12345/withdrawals",
"transfers": "/accounts/12345/transfers",
"close-requests": "/accounts/12345/close-requests"
}
}
}Here, we see the account entity has an extra links property mapping name the names and URLs that are available to the consumer. These would be computed on the server side. If, for example, the account had a negative balance the withdrawals link may not be present. The presence (or not) of the links are controlled by the state of the entity.
This approach is useful to my use case because I would be able to represent access control by showing or hiding actions based the presence of these links. It’s relatively small in terms of additional size to the JSON payload that is returned to the consumer, and it’s relatively simple to implement.
There are a couple of things that bother me with HATEOAS, though.
links as a property
This is completely a subjective opinion. Aesthetically, having a property named links foisted onto your JSON entities doesn’t feel quite right. What if you have an entity with a field named links ? What should you do in this case? You’d need to consider this early on and be consistent across all your services.
Links are just links
So I’ve provided a link back to the consumer. How do I use it? Is that a POST or a GET? Is it something else? Consider this example (adapted from my use case):
{
"property": {
"key": "foo",
"name": "Foo Property",
"description": "This is a property",
"value": "Foo!",
"links": {
"update": "/property/1",
"update-value": "/property/1",
"delete": "/property/1"
}
}
}The consumer has three actions available to them:
- Update the property name and description.
- Update the property value.
- Delete the property.
The URL for all these actions is identical, so how does their usage differ? To understand that these are PUT, PATCH, and DELETE operations respectively, you would need to refer to the API documentation.
HAL
Essentially, Hypertext Application Language (HAL) offers a straightforward format for creating links between your resources. These links contain a target URI and the name of the link, also known as rel, short for relationship. Furthermore, HAL provides a way embed related objects within an object. Here is another example shamelessly stolen from Wikipedia.
{
"_links": {
"self": {
"href": "http://example.com/api/book/hal-cookbook"
},
"next": {
"href": "http://example.com/api/book/hal-case-study"
},
"prev": {
"href": "http://example.com/api/book/json-and-beyond"
},
"first": {
"href": "http://example.com/api/book/catalog"
},
"last": {
"href": "http://example.com/api/book/upcoming-books"
}
},
"_embedded": {
"author": {
"_links": {
"self": {
"href": "http://example.com/api/author/shahadat"
}
},
"id": "shahadat",
"name": "Shahadat Hossain Khan",
"homepage": "http://author-example.com"
}
},
"id": "hal-cookbook",
"name": "HAL Cookbook"
}This shows the details of a book that is an element in a collection of books. The author is an embedded object containing a link to the author. Navigation of the collection is provided by the _links property. We can see the relationship of these links provide information as to what they do:
| Relationship | Function |
|---|---|
| self | The link to this book. This is probably what was hit in the first place to obtain this response. |
| next | The link to the next book in the collection. |
| previous | The link to the previous book in the collection. |
| first | The link to the first book in the collection. |
| last | The link to the last book in the collection. |
From what I can tell, the value you can give for a rel is not arbitrary. Wikipedia notes that HAL provides links, but not actions. Presumably you can’t say that your rel value is update. Though I don’t know that for sure. If there’s no enforcement I guess you can use it however you want.
Comparison of HATEOAS and HAL
On the surface, HATEOAS and HAL seem pretty similar. Both can show links, but HAL makes it clearer that links aren’t a natural part of the entity. They both label links; HATEOAS calls it a name, while HAL uses a relationship descriptor. That’s why I think HAL’s links are unsuitable for describing actions. Actions are not relationships. A plus for HAL is that it lets you describe relationships between entities, which is useful for giving more context and understanding how things are connected.
It’s a common criticism of HATEOAS and HAL that they add a lot of overhead to your service responses. I don’t have a particular issue with this, though I understand they can get quite large if you’re describing all the links and relationships in a long list of entities. I deal with large JSON payloads daily in my job. It hasn’t been a problem so far, but your mileage may vary. Another criticism is HATEOAS is more complicated to implement than HAL. I don’t really see this myself. to me they are just as complicated as each other. The complication probably stems from implementing them in a sane way.
So, if HATEOAS can describe actions to some extent but not relationships, and HAL can describe relationships but not actions, then neither is good enough for my use case. I need to find an alternative.
Introducing Siren
Amongst the other suggestions for adding context and intent to Web APIs was Structured Interface for Representing Entities, or Siren, which is what I’ve decided to use for my API design. Siren gives you structures to share information about entities, actions to change states, and links to help clients navigate. Let’s take a look at the example to see it in action:
{
"class": [ "order" ],
"properties": {
"orderNumber": 42,
"itemCount": 3,
"status": "pending"
},
"entities": [
{
"class": [ "items", "collection" ],
"rel": [ "http://x.io/rels/order-items" ],
"href": "http://api.x.io/orders/42/items"
},
{
"class": [ "info", "customer" ],
"rel": [ "http://x.io/rels/customer" ],
"properties": {
"customerId": "pj123",
"name": "Peter Joseph"
},
"links": [
{ "rel": [ "self" ], "href": "http://api.x.io/customers/pj123" }
]
}
],
"actions": [
{
"name": "add-item",
"title": "Add Item",
"method": "POST",
"href": "http://api.x.io/orders/42/items",
"type": "application/x-www-form-urlencoded",
"fields": [
{ "name": "orderNumber", "type": "hidden", "value": "42" },
{ "name": "productCode", "type": "text" },
{ "name": "quantity", "type": "number" }
]
}
],
"links": [
{ "rel": [ "self" ], "href": "http://api.x.io/orders/42" },
{ "rel": [ "previous" ], "href": "http://api.x.io/orders/41" },
{ "rel": [ "next" ], "href": "http://api.x.io/orders/43" }
]
}There is a lot of data there. Siren responses can be very verbose, but that makes them very expressive of context and intent. Let’s break it down.
- There is a
classproperty. This tells us what the entity is. In this case it’s an order. - The
propertiesproperty provides the entity’s data. In this case we have 3 properties: order ID, how many items are in the order, and the order’s status. - The
entitiesproperty provides a list of related entities. One of the features of Siren I like is that the order item entities is provided as a link that the JavaScript client can lazy load. We also have information about the customer who placed the order. - The
actionsproperty provides a list of state transitions that can be performed on the entity. This is a particular strength of Siren over HATEOAS I particularly appreciate as not only can you document the HTTP method for performing the action, but also the fields that are expected and the format the request will take. Here, we told exactly how to add an item to the order. - The
linksproperty provides relevant links. Here we are given links for navigating through a list of orders.
With Siren we can clearly define what data makes up an entity, what actions can be performed upon it and how, what other entities relate to the entity and provide other relevant links to the client. That is a full compliment of context and intent and should give pause for thought on how best we can communicate this type of data in our APIs. There is a schema too, so you can validate your responses. This is useful as your client library may reject invalid responses.
Introducing Problem+JSON
Problem+JSON is a standard for representing error messages in a JSON structure. I discovered Problem+JSON a couple of years ago and instantly wondered why I earth we keep reinventing the wheel when it comes to presenting errors and issues to the user. Look! It even has an RFC! And this is the second version, obsoleting a previous RFC that was published in 2016. Let’s take a look at an enhanced example from the RFC and dissect it:
HTTP/1.1 400 Bad Request
Content-Type: application/problem+json
{
"type": "https://example.net/validation-error",
"title": "Your request parameters didn't validate.",
"status": 400,
"detail": "Several request parameters were found to be invalid. Please check them and resubmit."
"invalid-params": [
{
"name": "age",
"reason": "must be a positive integer"
},
{
"name": "color",
"reason": "must be 'green', 'red' or 'blue'"
}
]
}- The
typefield is a URI that describes the problem type. A human-readable value is encouraged, such as a URL here. But it could also beurn:problems:validation-error. If it’s undefined then it’s assumed to beabout:blank. - The
titlefield provides a short description of the problem. - The
statusfield provides an HTTP status for the problem. - The
descriptionfield allows for a longer description of the problem. - Here,
invalid-paramsis a custom field. This is giving details of the particular fields that are invalid. Custom fields, or “extension members” as they are called in the RFC, may be used to describe additional information.
This is not the only way to represent this result. In fact, it’s not the way I would have done it. Let’s rewrite it a bit:
{
"type": "about:blank",
"title": "Validation failed.",
"status": 400,
"detail": "One or more request parameters were found to be in error. Please check them and resubmit.",
"errors": [
{
"type": "https://example.net/problem/positive-integer",
"title": "Must be a positive integer.",
"detail": "Age must be a positive integer.",
"field": "age"
},
{
"type": "https://example.net/problem/invalid-value",
"title": "Value is not a valid option.",
"detail": "Colour must be 'green', 'red' or 'blue'.",
"field": "colour"
}
]
}Here, we have nested the issues as problems themselves. Here they can take advantage of the Problem+JSON structure. I’ve omitted the status and kept the status 400 on the root problem entity. The specification does allow you to specify multiple statuses using the code 207:
HTTP/1.1 207 Multi Status
Content-Type: application/problem+json
{
"type": "about:blank",
"title": "Validation failed.",
"status": 207,
"detail": "One or more request parameters were found to be in error. Please check them and resubmit.",
"errors": [
{
"type": "https://example.net/problem/positive-integer",
"title": "Must be a positive integer.",
"status": 400,
"detail": "Age must be a positive integer.",
"field": "age"
},
{
"type": "https://example.net/problem/person-not-found",
"title": "Person not found.",
"status": 404,
"detail": "The person was not found.",
"field": "person_id"
}
]
}What I particularly like about Problem+JSON is the type field. While is doesn’t need to resolve to a real place, there is an encouragement to write and maintain your error documentation at those URLs. This is great for providing context to errors and more information on solving them.
Practical Examples
So, how did I choose to use these technologies? Well, probably incorrectly but I’ll illustrate it anyway. Firstly, Let me share a sample Siren payload. As a quick description of what is being described I’ll summarise it.
The payload describes a service that is located in a group (which could also be in a group). The service has two lists of related entities: properties and service keys. The properties entity list should not be confused with properties of an entity. Writing this out almost makes me think I’ve introduced confusion rather than avoided it.
The service can be updated or deleted. This is described in the actions property of the root service entity. We can see the precise URL and HTTP method and parameters to these calls. In addition to these calls we can also create a property or a service key. If any of these actions were not available to the user they would not be listed.
The related entities of properties and service keys are listed in their entirety. Due to how the application was designed they can be returned in one payload as they are not lazy loaded from the database. Each of these entities has its own set of actions that can be performed on it.
{
"actions": [
{
"fields": [
{
"name": "name",
"type": "text"
},
{
"name": "description",
"type": "text"
}
],
"name": "update-service",
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9",
"method": "PUT",
"title": "Update Service",
"type": "application/json"
},
{
"name": "delete-service",
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9",
"method": "DELETE",
"title": "Delete Service"
},
{
"fields": [
{
"name": "name",
"type": "text"
},
{
"name": "description",
"type": "text"
},
{
"name": "key",
"type": "text"
},
{
"name": "value",
"type": "text"
}
],
"name": "create-property",
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9/properties",
"method": "POST",
"title": "Create Property",
"type": "application/json"
},
{
"fields": [
{
"name": "name",
"type": "text"
},
{
"name": "description",
"type": "text"
},
{
"name": "create_property",
"type": "checkbox"
},
{
"name": "update_property",
"type": "checkbox"
},
{
"name": "list_properties",
"type": "checkbox"
}
],
"name": "create-service-key",
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9/service_keys",
"method": "POST",
"title": "Create Service Key",
"type": "application/json"
}
],
"class": [
"service"
],
"entities": [
{
"class": [
"breadcrumb",
"collection"
],
"links": [
{
"class": [
"group"
],
"href": "http://localhost:8080/v1/groups",
"rel": [
"self"
],
"title": "Home"
},
{
"class": [
"group"
],
"href": "http://localhost:8080/v1/groups/82b965a5-5fc3-486a-b097-cd081e72e128",
"rel": [
"self"
],
"title": "Group 1"
},
{
"class": [
"service"
],
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9",
"rel": [
"self"
],
"title": "Test Service"
}
],
"rel": [
"https://localhost:3000/docs/rels/breadcrumb-items"
]
},
{
"class": [
"properties",
"collection"
],
"entities": [
{
"actions": [
{
"fields": [
{
"name": "name",
"type": "text"
},
{
"name": "description",
"type": "text"
}
],
"name": "update-property",
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9/properties/f994fd12-dabb-4ba4-989c-2c64115444a7",
"method": "PUT",
"title": "Update Property",
"type": "application/json"
},
{
"fields": [
{
"name": "value",
"type": "text"
}
],
"name": "update-property-value",
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9/properties/f994fd12-dabb-4ba4-989c-2c64115444a7",
"method": "PATCH",
"title": "Update Property Value",
"type": "application/json"
},
{
"name": "delete-property",
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9/properties/f994fd12-dabb-4ba4-989c-2c64115444a7",
"method": "DELETE",
"title": "Delete Property"
}
],
"class": [
"info",
"property"
],
"links": [
{
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9/properties/f994fd12-dabb-4ba4-989c-2c64115444a7",
"rel": [
"self"
]
}
],
"properties": {
"name": "Test Property",
"key": "test",
"value": "Test",
"description": "Does nothing",
"id": "f994fd12-dabb-4ba4-989c-2c64115444a7"
},
"rel": [
"https://localhost:3000/docs/rels/property"
]
}
],
"rel": [
"https://localhost:3000/docs/rels/properties"
]
},
{
"class": [
"service_keys",
"collection"
],
"entities": [
{
"actions": [
{
"fields": [
{
"name": "name",
"type": "text"
},
{
"name": "description",
"type": "text"
},
{
"name": "create_property",
"type": "checkbox"
},
{
"name": "update_property",
"type": "checkbox"
},
{
"name": "list_properties",
"type": "checkbox"
}
],
"name": "update-service-key",
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9/service_keys/65d67eba-5745-48e6-92f3-a582d4cdff78",
"method": "PUT",
"title": "Update Service Key",
"type": "application/json"
},
{
"name": "delete-service-key",
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9/service_keys/65d67eba-5745-48e6-92f3-a582d4cdff78",
"method": "DELETE",
"title": "Delete Service Key"
}
],
"class": [
"info",
"service_key"
],
"links": [
{
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9/service_keys/65d67eba-5745-48e6-92f3-a582d4cdff78",
"rel": [
"self"
]
}
],
"properties": {
"id": "65d67eba-5745-48e6-92f3-a582d4cdff78",
"list_properties": true,
"description": null,
"update_property": false,
"name": "Simple Service",
"prefix": "EyPIweQUq5OY",
"create_property": false
},
"rel": [
"https://localhost:3000/docs/rels/service-key"
]
}
],
"rel": [
"https://localhost:3000/docs/rels/service-keys"
]
}
],
"links": [
{
"href": "http://localhost:8080/v1/services/f77d2a2d-493a-4b65-91e6-ca0f59652bd9",
"rel": [
"self"
]
}
],
"properties": {
"parent_group_id": "82b965a5-5fc3-486a-b097-cd081e72e128",
"id": "f77d2a2d-493a-4b65-91e6-ca0f59652bd9",
"name": "Test Service",
"description": "I am a test service"
}
}It makes for a large payload when formatted, but I think it’s worth it. Relations can be documented and hosted at the given URLs. All the context and intent is provided at every step of the way. This will ultimately make things easier for an integrator to understand and use the API.
Now let’s look at a simple validation error returned by the API.
{
"type": "about:blank",
"title": "Validation failed",
"status": 400,
"errors": [
{
"type": "https://localhost:3000/docs/problem/key-exists",
"title": "Key exists",
"status": 400,
"detail": "A property with the key 'test' already exists on service f77d2a2d-493a-4b65-91e6-ca0f59652bd9",
"field": "#/key"
}
]
}I have not linked a unique type for the root problem describing a validation failure. I would argue it’s not strictly necessary in this case as each problem listed does have a type. This type is a URL that could link to documentation about the problem and how to resolve it. The problem also describes the issue and points to the field in error. Again, all information is provided in a computer-parsable format.
Conclusion
Designing APIs with clear context and intent is important for easy use and integration. Using formats like Siren and Problem+JSON improve APIs with structured ways to show entities, links, actions, and errors. Using these formats well can greatly improve the developer experience and the overall effectiveness of an API. As we improve our API design practices, using these format will help create more robust, intuitive, and maintainable systems.
Banner image by Daria Glakteeva on Unsplash