Custom Headers vs Query String

2 Replies

I recently had a good discussion with my colleagues on a REST (piggybacking on  HTTP) service API design.  Quickly a very simple question piqued the interest of us all –  “what information should be requested in the (custom) headers and what information should be requested in the query string”.

Taking a step back, it is worth asking why this question would even arise? It is because both, http headers and query string parameters,  are valid input vehicles for a REST resource that influence the response it returns.

It is helpful to take one more step back to bring in two REST concepts to the fore to set up the context for our answer:

  • A single resource could be a collection of entities. Fielding defines “a resource R is a temporally varying membership function MR(t), which for time t maps to a set of entities, or values, which are equivalent.”
  • A REST ecosystem is a collection of resources.

We should examine a few characteristics (below) of the parameter under consideration to find a home for it:

To Restrict or To Offer Filters

As mentioned earlier, a resource is a collection of entities.  Both headers and query string parameters play a role in selecting a subset of entities from the collection.  If the need is to restrict or preselect a subset unbeknownst to the client, then headers should be used. On the other hand, if the need is to allow the client to choose a subset, then query string parameters must be used.

For example: The /orders is a resource of order entity collection in a store.

  • When a customer accesses /orders,  he/she is only presented with the orders he/she created.
  • However, when the store manager accesses /orders, he/she is presented with all the orders.

In the above case,  the resource pre-filtered the order entity collection by using the contextual information (store manager or customer).

Query strings can be used by both of them further narrow the order entities in this fashion /orders?date=today

  • The customer is presented with the orders he/she placed today.
  • The store manaager is presented with all the orders by all customers placed today.

In the above case, the user (store manager or customer) is specifically request a subset of the order entities.

More than filtering

The sweet spot for query string parameters is resource filtering (like the cases we talked about earlier). For other requirements  we need to start from headers and work towards query string parameters (i.e if we have to).

HTTP offers a wide variety of standard headers addressing a wide variety of needs:

  • Some, like Content-Type, describe the representation used.
  • Some others, like If-Modified-Since, influence resource caching.
  • And only a few, like Authorization and Range, could have an “effect” similar to filtering of collections of entities.

HTTP headers should be used to send contextual information about the request to the resource.

There is nothing in REST specification that limits us to the standard headers.  It should be fine to roll our own custom header in case of a specific justifiable need – HTTP itself is designed to be extensible.

Specificity to a resource

Query string parameters have a strong correlation to the properties of the entity behind a resource. For example:  “order_date” property belongs to a the order entity and it is a natural fit for a query string. This fits in nicely the filtering view of resources – obviously it logical to filter entities with the properties on it.

On the other hand, headers seldom have a direct correlation to specific properties of the entity behind a resource.

If the parameter under consideration has a strong affinity to a resource, then placing it in the header is not the right choice.

Sphere of influence

Query string’s specificity to a resource also limits it’s influence (so simply reusability) to a single source (very few times more than one).

In the cases where the parameter under consideration is expected to be applicable to all resources in the ecosystem, then locating this parameter in the header makes most sense. It is worth emphasizing that “being applicable” is not the same as mandatory.

Origination

Another hint one could use to determine if a parameter belongs to in the query string or the header is by asking who/when/where provides the value for the parameter.

Header parameters are typically added out-of-band by applications like user agents and/or proxies. If you need to allow message intermediaries to add the parameter to the message, then it should be located in the header.

In case of query string,  parameters can usually be traced back to some kind of direct input accepted by the system to satisfy a request. Query string remains untouched by intermediate proxy layers.

In other words, query string is reserved for end users/applications that are invoking resource. On the other hand headers are reserved for applications and intermediaries that are facilitating this invocation.

Finally,

At the core of it, to locate parameter in a header or a query is not an exact science. However, the above guidelines could help lending some objectivity to most decisions.

With Right for mimeType comes Responsibility too

Leave a reply

About a few weeks back, Amazon SWF started validating the content type of the incoming request. As a result, a my-until-then-working-application started failing with this exception:

{\”__type\”:\”com.amazon.coral.service#UnknownOperationException\”,\”message\”:null},\”Version\”:\”1.0\”}”

To get past the error, I had to set the contentType to  ‘application/x-amz-json-1.0’ instead of ‘application/json’.

Not a big deal (just a few hours of troubleshooting) until I put my REST glasses on (note: neither amazon swf nor I are claiming that amazon swf is REST compliant) .  This incident did give me a chance to stop and ponder over API design in general and REST API design in particular. The spirit of REST compliant architecture is to enable independent evolvability of resources (like Amazon SWF) and consumers (my application).  I believe these are a few points to bear in mind:

Describe your representation (mimeType)

Representations are the medium using which consumers modify a resource. Hence, if you roll a custom mimeType, it is your responsibility to provide a detailed definition of it, like application/json, or text/vcard.

Amazon did not do that with ‘application/x-amz-json-1.0’.  Currently, i am not sure what it is. You and I can guess what is, but that is not how things should be.

Further, REST does not specify where your custom representation definition should be stored. You could create your own “well known” location, like IANA, where your consumers can look it up.

Don’t “alias” existing mimeTypes

REST is all about relying on standards, on the premise that it will reduce dependency. If there is an existing standard that fits your needs, like application/json, then use it. It is not being responsible to roll out a mimeType that is exactly like an existing one. It appears that you might be hedging against changes to json or preparing to roll your own. In either case, your are guaranteed to break your client in future, because your client and you are not working of the same standard.

Roll your own mimeType, only when it has something special to offer. mimeTypes/representations (reading/writing) are “generic” skills that your client needs to master. For example, browsers need to know how to render ‘text/html’, ‘image/jpeg‘ and so on and so forth.

Preserve backward compatibility

This is the most obvious one. Find a way to keep your client up and running, especially for such seemingly superficial change like replacing ‘application/json’ with ‘application/x-amz-json-1.0”

In search of REST

Leave a reply

I believe there is hardly any developer these days who has not heard of REST (Representational State Transfer), and many have worked with it in some shape or form.  And more amazingly, there is probably no one this planet who is not positively affected by this architecture. I fall into the dabbled-with-some-cloud-apps camp.  For a long time, I have been uncomfortable with how a (seemingly) tight coupling exists between REST and HTTP.  I wanted to find out if that is indeed a fair statement.  A  geekation was in order.

To understand REST, I started right where it all started, Roy Fielding’s dissertation paper – chapter five.  A 30,000 foot level view of REST  is good place to start too.

REST is an architectural style to connect distributed systems (REST calls them connectors). Historically, software designs and architecture styles have mirrored patterns and practices in the real world. I expect the same to hold true for REST too. To understand and appreciate the strengths of REST, I will apply it’s principle to analog worlds. The example, I picked for applying REST principles is “banking”.  A client will interact with his bank in a local branch (we can draw parallel’s to a client application interacting with a REST service in the digital world).

Note: In most of this article, I rephrased many of REST’s terminology to make it readable. However, I provided a link to the official terminology as much as possible.

Resource

The foundational building block in a REST compliant architecture is a resource.  In the simplest terms, anything that can be looked up by a name could be considered a resource. It could be information or a service. In our example, the resources could be people like  “teller”,  things like “savings account # 123”, “checking account # 456”. And possibly services offered by the bank like “current balance”, “deposit funds” , “withdraw funds”.

The fundamental philosophy of REST is to standardize (within a context – for us it the bank) and minimize assumptions to the extent that resources and their consumers can evolve in vacuum (strong word, but you get the point) but still integrate (in real time) when needed.  REST achieves this by requiring all connectors to conform to an uniform interface.

Uniform Interface

The term interface has a special meaning to us developers. REST uses interface to mean something different when its says uniform interface.  It is not a strict and complete specification of behaviors that a connector will have. Rather, it is a set of (minimum) constraints that the connecters must satisfy.

Constraint # 1: Resources must have identifiers

A resource is known by its resource identifier. It is a unique and and unambiguous name of a resource within a REST ecosystem.

A few examples of resource identifiers are:

  1. savings account # 123
  2. checking account # 456
  3. And possibly…
    • current balance of account #123
    • deposit funds into account #123
    • withdraw funds from account #123

A resource itself is encapsulated by its provider.  A bank’s client will never know how the bank internally represents “checking account # 456” resource. Only the resource identifier, the account number # 456,  is exposed to the client. . Here are some features of resource identifiers:

  1. Each resource must be identifiable by an unique  identifier.
  2. The identifier should be immutable.
  3. There is no rigid syntax for identifiers. However, URI is a widely accepted REST compliant identifier scheme.
  4. Some resources are “well known” and some need to be “discovered” (covered by a later constraint # 5)

It is important to emphasis that a resource (identifier) is very different from an entity (identifier). Multiple resources, (for example: “deposit funds into account # 123” and  “withdraw funds from account # 123”) could manipulate a single entity (for example: account  # 123).

Constraint # 2: All resources in a portfolio must bear the same interface “commands”

The client expects to the following with his/her account:

Commands … applicable to these resources
  1. Withdraw money
  2. Deposit money
  3. Current balance
  4. Request statement
  5. Help
  1. Savings Account
  2. Checking Account

* I provided the Help command, so that the client can learn how to or what is needed to successfully work with a resource. REST does not suggest we provide this, but I think it is an important cog to further client self sufficiency and decoupling.

Contextual commands means coupling and dependency. REST eliminates this by mandating that all resources respond to the same commands, however, their behavior could vary wildly. This is similar (only slightly) to polymorphic behavior in object oriented design.

However, there is a tension between the variety of resources in the ecosystem and how many “commands” we can standardize on. The more resource variety we to the ecosystem,  fewer uniform commands we can afford. For example, if we added a semantically different resource ( like a credit card) to our bank’s resource portfolio, it will have dramatic affect on the resource-commands balance. This is because, a client can increase his credit limit, which he/she cannot do with a savings or checking account.

Commands … applicable to these resources
  1. Do
  2. Help
  1. Increase limit of credit card
  2. Current balance of savings account
  3. Current balance of checking account
  4. Current balance of credit card
  5. Deposit from savings account
  6. Deposit from checking account
  7. Deposit from credit card
  8. Request statement of savings account
  9. Request statement of checking account
  10. Request statement of credit card
  11. Witdraw from savings account
  12. Withdraw from checking account
  13. Withdraw from credit card

REST does not specify what the uniform commands should be offered, it only requires that all resources have the same set of commands.  It is up to us evaluate our domain and come up with a reasonable and static list.

Constraint #3: Messages must be self descriptive

Clients and resources communicate with each other using messages.

For example, “I need to deposit 200.00 USD. I will pay by a cheque“, is a message the client would give to the “savings account # 123” resource.

This constraint helps the client & resource to choose a mutually convenient message structure for communication. This further eliminates assumptions and decouples the two systems.  REST breaks down a message into three parts, viz.,

Example Message Component Role
200.00 USD Data This is the soul of the message. Resources/clients act on this.
Cheque Metadata  In the spirit of eliminating assumption, the component describe how the recipient (resource or client) should interpret the data.  More on this in constraint # 4.
Deposit Control data  How the client intends to modify the resource. See constraint # 2.

Constraint # 4:  Use “representations” to manipulate resources

Our client has a few possible ways to give 200.00 USD to the bank. Choices could be:

  1. 2 Benjamins
  2. 40 Lincolns
  3. 200 Washingtons
  4. 20,000 cents
  5. A cashiers cheque

These are how the client could get money into or out of say “checking account # 456”.

Some representations, even though possible, may not be acceptable to one party or both parties.  For example, your bank might refuse to accept 20,000 cents, and might request to get the money in 2 Benjamins, but your friend might want 200 Washingtons.

A few things to pay attention to:

  1. Representations lend concreteness to the abstract resources.
  2. Representations can be chosen on a per transaction basis.
  3. Representations are reusable, to work with more that one resource. In our example, they can be used to “deposit into checking account # 456” and “withdraw from checking account # 456”.

Constraint #5:  Resources are progressively discoverable

In a RESTful eco-system, some resource identifiers are like  the “north star” – guaranteed to be there for clients (for example, a teller in a bank). The moment you walk into the bank, you expect find a teller.  There will (and should) be very few resource identifiers of this kind.

On the other hand, many resource identifiers are ephemeral – new ones created, old ones destroyed or moved around (for example, the products offered by a bank,  customers accounts). This constraint is about not coupling the clients to these ephemeral resources identifiers. Instead, rely on the north-star resource identifiers to guide the clients to their desired ephemeral resource identifiers.  For example, the teller could tell you about the bank’s money market accounts and give your the resource identifier that can help you create one.  This type of organization of resource identifiers is called hypermedia.  The most well known example of  hypermedia is the “world wide web” .

A few things to pay attention to:

  1. Resource include resource identifiers to other contextual resource identifies in its response message.
  2. Once the client receives these resource identifiers, it will use the resource’s standard interface commands to work with it.

References on this constraint

  1. http://www.peej.co.uk/articles/hypermedia-as-the-engine-of-application-state.html
  2. http://blog.programmableweb.com/2012/08/01/rest-hypermedia-and-the-future-of-enterprise-web-apis/
  3. http://blueprintforge.com/blog/2012/01/01/a-short-explanation-of-hypermedia-controls-in-restful-services/
  4. http://www.slideshare.net/trilancer/why-hateoas-1547275
  5. http://www.infoq.com/articles/mark-baker-hypermedia
  6. Distributed Hypermedia

Post Geekation Blues

Like any good architecture, REST provides facility for various implementations. REST based on HTTP is the most common pattern, but does not have to the only pattern.  This coupling is so prevalent that even Fielding is frustrated with it.

Amazon: Simple Workflow

Leave a reply

It is amazing how much free time we have after the football season ends :-).  My football season ended two weeks ago when the New England Patriots were eliminated (it still hurts – Next year will be different).  I made good use of this additional time by taking a geekation that had been wanting to for a long time.

I decided to take a trip to Amazon’s SWF land.  I have been hearing a lot about it, at work,in the community  After all, it is on THE CLOUD.

With destination picked, I needed a theme to make the most of my stay there. I chose to build a very rudimentary digital asset management system. The idea is to touch upon the major moving parts for a digital asset management system and focus on the happy path. I eventually hope to evolve this into a reference implementation. I don’t expect anyone (including myself) to put this code in production. Just have some fun and learn.

I selected these cloud solutions to build my reference application on.

  1. Amazon SWF
  2. Amazon S3
  3. Amazon RDS
  4. Amazon CloudSearch
  5. Encoding.com

The idea is figure out how to stitch these cloud solutions together.

On the back of a paper napkin, the activity workflow to upload a video file into a digital asset management system could look like this:

Activity

To implement the above workflow in the SWF framework, I needed to build three classes of applications:

  1. Activity Workers
  2. Workflow Deciders
  3. Workflow Triggers

Activity Worker

An Activity Worker is an application that hosts logic to perform an activity (in the above diagram).  It pulls work from an SWF task queue (called task list), works on it and finally reports the result back to SWF.

  • Typically, one Activity Worker does one Activity. This is ideal for several reasons.
  • In a few rare cases, one worker can handle more than one activity.

Since workers in SWF worker solely on “pull” (asynchronous) model, three scalability options are available to us, viz.,

  • Demand smoothening; a worker chooses work that matches its available capacity.
  • Scale out; stand up more instances of workers as demand increases.
  • Scale up; throw more hardware at a single worker.

A mentioned earlier, a worker pulls activity off task list in SWF.

  • Typically, a one-to-one relationship exists between a task list and an activity type.
  • Alternatively (but rarely),
    • A task list can contain more than one activity type. A case where a single worker can handle more than one activity.
    • A single activity type (not activity instance) can appear more than one task list. A case where activity instances need to be prioritized.

Workers typically, like I did, are implemented as daemon processes. However, there is nothing in the SWF architecture that prevents from deciders and workers being interactive application like web application or even console applications.

The activity workers I have in my application are:

  1. Upload Worker : Responsible for uploading the digital files into the DAM.
  2. Transcode Worker: Responsible for creating variation(s) for the uploaded digital file (Uses Encoding.com)
  3. Asset Management Worker : Responsible for recording the metadata of uploaded digital file.
  4. Asset Index Worker : Responsible for indexing the metadata in a search engine (Amazon CloudSearch)

To leverage all the benefits of SWF (asynchronous activities), we need to achieve the following in our design:

  1. We only assume the order in which activities are given to workers.
  2. We cannot assume the order in which activities will be completed (As one worker might be slower than the other)
  3. The activities on a task list should not have any dependency of each other. This will severely limit how we can scale our application.
  4. The workers should be idempotent.

Workflow Deciders

While workers do heavy lifting, deciders orchestrate the workers. They decide when an activity should be done. A decider also sets the policy for an activity that is enforced by SWF.

In many respects, deciders are like workers. They, like workers, pull decision tasks (created by SWF) from a task list, take decision, and report the result back to SWF.

Deciders can afford to be stateless. SWF, as part of the decision task request provides a workflow event log. By retracing the log the decider can figure out what should happen next. This type of decision-making could get tricky quite quickly, but that is the fun part.

I believe it is possible to scale-out the deciders, but we need to be extremely careful with our design. Imagine, two decider instances making a decision on the same workflow at the same time. Fortunately, SWF will give different workflow event log to both and workflow event logs are strictly sequential and append only. Now it is up to us to design the deciders to be idempotent.

I have two deciders:

  1. Single Submission Decider : A workflow to control the submission of a single digital file
  2. Bulk Submission Decider : A workflow to control a bulk submission. This piggy backs on the Single Submission Decider.

Workflow Triggers

Workflow triggering applications are usually consumer-facing applications that accept work and set the “workflow” ball rolling. Unlike workers and deciders, messages are pushed to these triggering applications.

I have one triggering application:

  1. Submission Management Service : A REST service that accept bulk submission requests and triggers the “Bulk Submission Workflow”
  2. Bulk Submission App : A command line application installed on the end user’s computer to submit digital assets. This consumes the Submission Management Service’s API.

What’s left for SWF to do?

Good question, right?

The right question to ask is “what we didn’t have to do build a distributed, scalable and synchronous digital asset management system”.

  1. Durable Task Queues
  2. Task Timeout
  3. Task Retries
  4. Workflow Traceability
  5. Policy based control of activities

Post Geekation Blues

This was good geekation.  Why to do I think so? Because I already a have list of “things to check out” on my next break

  1. Enhance the reference implementation to handle unhappy paths
  2. Can the workflow logic be externalized and put in the hand of business users?
  3. Project the operating cost of this reference implementation in some hypothetical business settings.