HTTP API

jim stafford

Introduction

Goals

The student will learn:

how the WWW defined an information system capable of implementing system APIs
identify key differences between a truly RESTful API and REST-like or HTTP-based APIs
how systems and some actions are broken down into resources
how web interactions are targeted at resources
standard HTTP methods and the importance to use them as intended against resources
individual method safety requirements
value in creating idempotent methods
standard HTTP response codes and response code families to respond in specific circumstances

Objectives

At the conclusion of this lecture and related exercises, the student will be able to:

identify API maturity according to the Richardson Maturity Model (RMM)
identify resources
define a URI for a resource
define the proper method for a call against a resource
identify safe and unsafe method behavior
identify appropriate response code family and value to use in certain circumstances

World Wide Web (WWW)

an information system of web resources
- identified by Uniform Resource Locators (URLs)
  - that can be interlinked via hypertext
  - transferred using Hypertext Transfer Protocol (HTTP). [1]
Web resources started out being primarily documents to be created, downloaded, replaced, and removed
- has progressed to be any identifiable thing — whether it be
  - the entity (e.g., person)
  - something related to that entity (e.g., photo)
  - an action (e.g., change of address) [2]

Example WWW Information System

standard set of content types
accessed through a standard set of methods
related through location-independent links using URLs

REST

Representational State Transfer (REST)
an architectural style for creating web services
- web services conforming to this style are considered "RESTful" web services [3]
defined in 2000 by Roy Fielding in his doctoral dissertation
- was also used to design HTTP 1.1. [5]
relies heavily on concepts and implementations used in the World Wide Web
- centers around web resources addressable using URIs

HATEOAS

at the heart of REST is the notion of hyperlinks to represent state
- called "Hypermedia As The Engine Of Application State" (HATEOAS)
- example:
  - presence of address_change link may mean the address of a person can be changed and client accessing that person representation is authorized to initiate change
  - presence of current_address and addresses links identifies how client can obtain current and past addresses for the person

interface contract allows
- clients dynamically determine current capabilities of resource
- resource dynamically add capabilities over time

Clients Dynamically Discover State

HATEOAS permits the capabilities of client and server to advance independently through the dynamic discovery of links. [6]

Figure 3. Example of Clients Dynamically Discovering State Represented through Hyperlinks

Static Interface Contracts

remote procedure call (RPC) techniques define static interface contracts
documented in detail to represent a certain level of capability offered by the server and understood by the client
capability change rollout under RPC approach may require coordination between all clients involved

Figure 4. RPC Tight Coupling Example

Internet Scale

As clients morph from

a few, well known sources
to millions of lightweight apps running on end-user devices

the need to decouple service capability deployments through dynamic discovery becomes more important

Do you have control of when clients update?

Design interfaces, clients, and servers with forward and backward compatibility in mind to allow for flexible rollout with minimal downtime.

How RESTful?

open and interfacing concepts of WWW are attractive to today’s API designers
however, implementing dynamic discovery is difficult
- potentially making systems more complex and costly to develop
REST officially contains more than most interface designs use or possibly need to use
- causing developments to take only what they need
- triggers some common questions:

"What is your definition of REST?"

"How RESTful are you?"

Buzzword Association

Given

for many developers and product advertisements eager to get their names associated with a modern and successful buzzword
- REST is (incorrectly) anything using HTTP that is not SOAP
for others
- REST is (still incorrectly) anything that embraces much of the WWW but still lacks the rigor of making the interfaces dynamic through hyperlinks

Then

places us in state where much of world refers to something as REST and RESTful
when what they have is far from the official definition

REST-like or HTTP-based

Giving a nod to this situation, we might use a few other terms:

REST-like
HTTP-based

Better yet and for more precise clarity of meaning, I like the definitions put forward in the Richardson Maturity Model (RMM).

Richardson MaturityModel (RMM)

Richardson Maturity Model (RMM) was developed by Leonard Richardson
- breaks down levels of RESTful maturity [7]

Level 3

using Hypermedia Controls i.e., the basis for Roy Fielding’s definition of REST
dynamic discovery of state and capabilities thru hyperlinks

Level 2

using HTTP Methods i.e., handle similar situations in the same way
standardized methods and status codes
publicly expose method performed and status responses to better enable communication infrastructure support

Level 1

using Resources i.e., divide and conquer
e.g., rather than a single aggregate for endpoint calls, make explicit reference to lower-level targets

Level 0

using HTTP soley as a transport
e.g., CORBA tunneled all calls through HTTP POST

"REST-like"/"HTTP-based" APIs

Common "REST-like" or "HTTP-based" APIs are normally on a trajectory to strive for RMM Level 2 and are based on a few main principals included within the definition of REST.

HTTP Protocol
Resources
URIs
Standard HTTP Method and Status Code Vocabulary
Standard Content Types for Representations

Uncommon REST Features Adopted

Links are used somewhat
- rarely used in an opaque manner
- rarely used within payloads
- rarely used with dynamic discovery
Clients
- commonly know the resources they are communicating with ahead of time
- build URIs to resources based on exposed details of API and IDs returned in earlier responses

That is technically not a RESTful way to do things.

RMM Level 2 APIs

Although I will commonly hear projects state that they implement a "REST" interface (and sometimes use it as "HTTP without SOAP"), I have rarely found a project that strives for dynamic discovery of resource capabilities as depicted by Roy Fielding and categorized by RMM Level 3.

These APIs try to make the most of HTTP and the WWW, thus at least making the term "HTTP-based" appropriate and RMM-level 2 a more accurate description. Acknowledging that there is technically one definition of REST and very few attempting to (or needing to) achieve it — I will be targeting RMM Level 2 for the web service interfaces developed in this course and will generically refer to them as "APIs".

At this point lets cover some of the key points of a RMM Level 2 API that I will be covering as a part of the course.

technically one definition of REST
- very few attempting to (or needing to) achieve it
rarely will find projects striving for dynamic discovery of resource capabilities as depicted by Roy Fielding and categorized by RMM Level 3
- at best - these APIs try to make the most of HTTP and the WWW
- term "HTTP-based" is appropriate
- term RMM-level 2 more accurate
I will be targeting RMM Level 2 for the web service interfaces developed in this course
- will generically refer to them as "APIs"

At this point lets cover key points of a RMM Level 2 API that will be covered as a part of the course

HTTP Protocol Embraced

Various communications protocols have been transport agnostic. If you are old enough to remember SOAP , you will have seen references to it being mapped to protocols other than HTTP (e.g., SOAP over JMS) and its use of HTTP lacked any leverage of WWW HTTP capabilities.

For SOAP and many other RPC protocols operating over HTTP — communication was tunnelled though HTTP POST messages, bypassing investments made in the existing and robust WWW infrastructure. For example, many requests for the same status of the same resource tunnelled thru POST messages would need to be answered again-and-again by the service. To fully leverage HTTP client-side and server-side caches, an alternative approach of exposing the status as a GET of a resource would save the responding service a lot of unnecessary work and speed up client.

REST communication technically does not exist outside of the HTTP transport protocol. Everything is expressed within the context of HTTP, leveraging the investment into the world’s largest information system.

various communications protocols have been transport agnostic
- SOAP
  - mapped to protocols other than HTTP (e.g., JMS)
  - use of HTTP lacked any leverage of WWW HTTP capabilities
- communications tunnelled though HTTP POST messages
  - example: many requests for the same status of the same resource tunnelled thru POST messages would need to be answered again-and-again by the service
  - bypasses investments made in existing and robust WWW infrastructure

REST communication technically does not exist outside of HTTP transport protocol
- everything expressed within the context of HTTP
- leverages investment into the world’s largest information system
- example: expose resource status as a response GET of resource — would save the responding service a lot of unnecessary work and speed up client
- fully leverages HTTP client-side and server-side caches

Resource

service domains broken down into key areas, known as resources
- key characteristic of RMM Level 1 compliance
- primarily noun-based (e.g., Documents, People, Companies), lower-level properties, or relationships
- also include actions or a long-running activity
  - to be able to initiate them, monitor their status, and possibly perform some type of control
- nearly anything can be made into a resource
limited number of HTTP methods but can have unlimited resources
- examples:
  - products
  - categories
  - customers
  - todos

Nested Resources

Resources can be nested under parent or related resources.

categories/{id}
categories/{id}/products
todos/{name}
todos/{name}/items

Uniform Resource Identifiers (URIs)

Resources are identified using Uniform Resource Identifier (URIs).

URIs have a generic syntax with precise rules
- composed of several components
- specialized by individual schemes (e.g., http, mailto, urn)
- precise generic URI and scheme-specific rules guarantee uniformity of addresses

Example URIs

https://github.com/spring-projects/spring-boot/blob/master/LICENSE.txt#L6 (1)
mailto:joe@example.com?cc=bob@example.com&body=hello (2)
urn:isbn:0-395-36341-1 (3)

1	example URL URI
2	example email URI [9]
3	example URN URI; "isbn" is a URN namespace [8]

URI Generic Syntax

URI components listed in hierarchical significance — from left to right

Generic URI components [10]

URI = scheme:[//authority]path[?query][#fragment]

Authority Subcomponents [10]

authority = [userinfo@]host[:port]

Table 1. Generic URI Components
Scheme	sequence of characters, beginning with a letter followed by letters, digits, plus (+), period, or hyphen(-)
Authority	naming authority responsible for the remainder of the URI
User	how to gain access to the resource (e.g., username) - rare, authentication use deprecated
Host	case-insensitive DNS name or IP address
Port	port number to access authority/host
Path	identifies a resource within the scope of a naming authority. Terminated by the first question mark ("?"), pound sign ("#"), or end of URI. When the authority is present, the path must begin with a slash ("/") character
Query	indicated with first question mark ("?") and ends with pound sign ("#") or end of URI
Fragment	indicated with a pound("#") character and ends with end of URI

URI Component Examples

Example URL URI Components

           -- authority                                    fragment --
          /                                                           \
https://github.com/spring-projects/spring-boot/blob/master/LICENSE.txt#L6
\                  \
 -- scheme          -- path

Path cannot begin with two slash ("//") characters when authority is not present

Example mailto URI Components

         -- path
        /
mailto:joe@example.com?cc=bob@example.com&body=hello
\                       \
 -- scheme               -- query

Example URN URI Components

 -- scheme
/
urn:isbn:0-395-36341-1
    \
     -- path

URI Characters and Delimiters

URI characters are encoded using UTF-8
component delimiters: slash ("/"), question mark ("?"), and pound sign ("#")
many other special characters reserved for sub-components delimiters

Ref: URI Generic Schema

scheme:[//[userinfo@]host[:port]]path[?query][#fragment]

Reserved Generic URI Delimiter Characters

: / @ [ ] ? (1)

1	square brackets("[]") are used to surround newer (e.g., IPv6) network addresses

Reserved Sub-delimiter Characters

! $ & ' ( ) * + , ; =

Unreserved URI Characters

alpha(A-Z,a-z), digit (0-9), dash(-), period(.), underscore(_), tilde(~)

URI Percent Encoding

(Case-insensitive) percent encoding used to represent characters reserved for delimiters or other purposes
- (e.g., %x2f and %x2F both represent slash ("/") character)
unreserved characters should not be encoded

Example Percent Encoding

https://www.google.com/search?q=this+%2F+that (1)

1	slash("/") character is Percent Encoded as `%2F`

URI Case Sensitivity

generic components like scheme and authority are case-insensitive but normalize to lowercase
other components of URI are assumed to be case-sensitive

Example Case Sensitivity

HTTPS://GITHUB.COM/SPRING-PROJECTS/SPRING-BOOT (1)
https://github.com/SPRING-PROJECTS/SPRING-BOOT (2)

1	value pasted into browser
2	value normalized by browser

URI Reference

reference a target URI without specifying the complete URI
- can be the full target URI or a relative reference
relative reference allows
- resources to reference one another without specifying a scheme or upper parts of the path
- entire resource trees to be relocated without having to change relative references between them

Example Relative References

https://github.com/spring-projects/spring-boot/blob/master/LICENSE.txt#L6
//github.com/spring-projects/spring-boot/blob/master/LICENSE.txt#L6
/spring-projects/spring-boot/blob/master/LICENSE.txt#L6
spring-boot/blob/master/LICENSE.txt#L6
LICENSE.txt#L6
../master/LICENSE.txt#L6
#L6

URI Reference Terms

target uri

the URI being referenced

Example Target URI

https://github.com/spring-projects/spring-boot/blob/master/LICENSE.txt#L6

network-path reference

relative reference starting with two slashes ("//"). My guess is that this would be useful in expressing a URI to forward to without wishing to express http versus https (i.e., "use the same scheme used to contact me")

Example Network Path Reference

//github.com/spring-projects/spring-boot/blob/master/LICENSE.txt#L6

absolute-path reference

relative reference that starts with a slash ("/"). This will be a portion of the URI that our API layer will be well aware of.

Example Absolute Path Reference

/spring-projects/spring-boot/blob/master/LICENSE.txt#L6

URI Reference Terms (cont)

relative-path reference

relative reference that does not start with a slash ("/"). First segment cannot have a ":" — avoid confusion with scheme by prepending a "./" to the path. This allows us to express the branch of a tree from a point in the path.

Example Relative Path Reference

spring-boot/blob/master/LICENSE.txt#L6
LICENSE.txt#L6
../master/LICENSE.txt#L6

same-document reference

relative reference that starts with a pound ("#") character, supplying a fragment identifier hosted in the current URI.

Example Same Document Reference

#L6

base URI

leading components of the URI that form a base for additional layers of the tree to be appended

Example Base URI

https://github.com/spring-projects
/spring-projects

URI Naming Conventions

URI specifications do not list path naming conventions
REST promotes opaque URIs
however, common practice to name resource collections with URI path ending with plural noun

Example Resource Collection URI Absolute Path References

/api/products (1)
/api/categories
/api/customers
/api/todo_lists

1	URI paths for resource collections end with a plural noun

URI Naming Conventions (cont.)

Individual resource URIs are identified by an external identifier below the parent resource collection.

Example Individual Resource Absolute URI Paths

/api/products/{productId} (1)
/api/categories/{categoryId}
/api/customers/{customerId}
/api/customers/{customerId}/sales

1	URI paths for individual resources are scoped below parent resource collection URI

Nested resource URIs are commonly expressed as resources below their individual parent.

Example Nested Resource Absolute URI Paths

/api/products/{productId}/instructions (1)
/api/categories/{categoryId}/products
/api/customers/{customerId}/purchases
/api/todo_lists/{listName}/todo_items

1	URI paths for resources of parent are commonly nested below parent URI

URI Variables

query at the end of the URI path can express optional and mandatory arguments
- commonly used in queries

Query Parameter Example

http://127.0.0.1:8080/jaxrsInventoryWAR/api/categories?name=&offset=0&limit=0
name   => (null) #has value null
offset => 0
limit  => 0

nested path path parameters may express mandatory arguments

Path Parameter Example

http://127.0.0.1:8080/jaxrsInventoryWAR/api/products/{id}
http://127.0.0.1:8080/jaxrsInventoryWAR/api/products/1
id => 1

Methods

HTTP contains a bounded set of (9) methods
- represent "verbs" of what we are communicating relative to resource
bounded set provides a uniform interface across all resources
four primary methods you will see in most tutorials, examples, and application code

Table 2. Primary HTTP Methods
GET	obtain a representation of resource using a non-destructive read
POST	create a new resource or tunnel a command to an existing resource
PUT	create a new resource with having a well-known identity or replace existing
DELETE	delete target resource

Example: Get Product ID=1

GET http://127.0.0.1:8080/jaxrsInventoryWAR/api/products/1

Additional HTTP Methods

There are two additional methods useful for certain edge conditions implemented by application code.

HEAD

logically equivalent to a GET without response payload - metadata only. Can provide efficient way to determine if resource exists and potentially last updated

PATCH

partial replace. Similar to PUT, but indicates payload provided does not represent the entire resource and may be represented as instructions of modifications to make. Useful hint for intermediate caches

post providing role of PATCH and JSON Patch/JSON Merge Patch instructions

There are three more obscure methods used for debug and communication purposes.

Table 4. Communication Support Methods
OPTIONS	generates a list of methods supported for resource
TRACE	echo received request back to caller to check for changes
CONNECT	used to establish an HTTP tunnel — to proxy communications

Method Safety

proper execution of the internet protocols relies on proper outcomes for each method
with potential of client-side proxies and server-side reverse proxies in the communications chain — one needs to pay attention to what can and should not change the state of a resource
"Method Safety" is a characteristic used to describe whether a method executed against a specific resource modifies that resource or has visible side effects
- Safe — resource unmodified and no visible side effects
- Unsafe - resource modified or visible side effects

Safe and Unsafe Methods

The following methods are considered "Safe" — thus calling them should not modify a resource and will not invalidate any intermediate cache.

GET
HEAD
OPTIONS
TRACE

The following methods are considered "Unsafe" — thus calling them is assumed to modify the resource and will invalidate any intermediate cache.

POST
PUT
PATCH
DELETE
CONNECT

Violating Method Safety

Do not violate default method safety expectations

Internet communications is based upon assigned method safety expectations. However, these are just definitions. Your application code has the power to implement resource methods any way you wish and to knowingly or unknowingly violate these expectations. Learn the expected characteristics of each method and abide by them or risk having your API not immediately understood and render built-in Internet capabilities (e.g., caches) useless. The following are examples of what not to do:

Example Method Safety Violations

GET /jaxrsInventoryWAR/api/products/1?command=DELETE (1)
POST /jaxrsInventoryWAR/api/products/1 (2)
  content: {command:'getProduct'}

1	method violating GET Safety rule
2	unsafe POST method tunneling safe GET command

Idempotent

Idempotence - a characteristic where a repeated event produces same outcome every time executed
- very important concept in distributed systems that commonly have to implement eventual consistency
  - where failure recovery can cause unacknowledged commands to be executed multiple times
idempotent characteristic is independent of method safety
- only requires the same result state be achieved each time called

Idempotent and non-Idempotent Methods

The application code implementing the following HTTP methods should strive to be idempotent.

GET
PUT
DELETE
HEAD
OPTIONS

The following HTTP methods are defined to not be idempotent.

POST
PATCH
CONNECT

Relationship between Idempotent and browser page refresh warnings?

The standard convention of Internet protocol is that most methods except for POST are assumed to be idempotent. That means a page refresh for a page obtained from a GET gets immediately refreshed and a warning dialogue is displayed if it was the result of a POST.

Response Status Codes

each HTTP response is accompanied by a standard HTTP response code
- tells the caller whether the request succeeded or failed
- additional indications of a types if success or failures.
status codes are seperated into five (5) categories
- 1xx - informational responses
- 2xx - successful responses
- 3xx - redirected responses
- 4xx - client errors
- 5xx - server errors

Common Response Status Codes

Code	Name	Meaning
200	OK	"We achieved what you wanted - may have previously done this"
201	CREATED	"We did what you asked and a new resource was created"
202	ACCEPTED	"We received your request and will begin processing it later"
204	NO_CONTENT	"Just like a 200 with an empty payload, except the status makes this clear"
400	BAD_REQUEST	"I do not understand what you said and never will"
401	UNAUTHORIZED	"We need to know who you are before we do this"
403	FORBIDDEN	"We know who you are and you cannot say what you just said"
404	NOT_FOUND	"We could not locate the target resource of your request"
422	UNPROCESSABLE_ENTITY	"I understood what you said, but you said something wrong"
500	INTERNAL_ERROR	"Ouch! Nothing wrong with what you asked for or supplied, but we currently have issues completing. Try again later and we may have this fixed."

Code

Name

Meaning

200

"We achieved what you wanted - may have previously done this"

201

CREATED

"We did what you asked and a new resource was created"

202

ACCEPTED

"We received your request and will begin processing it later"

204

NO_CONTENT

"Just like a 200 with an empty payload, except the status makes this clear"

400

BAD_REQUEST

"I do not understand what you said and never will"

401

UNAUTHORIZED

"We need to know who you are before we do this"

403

FORBIDDEN

"We know who you are and you cannot say what you just said"

404

NOT_FOUND

"We could not locate the target resource of your request"

422

UNPROCESSABLE_ENTITY

"I understood what you said, but you said something wrong"

500

INTERNAL_ERROR

"Ouch! Nothing wrong with what you asked for or supplied, but we currently have issues completing. Try again later and we may have this fixed."

Representations

resources may have multiple independent representations
- no direct tie between data format received from clients, returned to clients, or managed internally
representations are exchanged using standard MIME or Media types
common media types for information include
- application/json
- application/xml
- text/plain
common data types for raw images include
- image/jpg
- image/png

Content Type Headers

Table 5. HTTP Content Negotiation Headers
Accept	defines a list of media types the client understands, in priority order
Content-Type	identifies the format for data supplied in the payload

Example Accept and Content-Type Headers

> POST /greeting/hello
> Accept: application/xml,application/json
> Content-Type: text/plain
hi

< 200/OK
< Content-Type: application/xml
<greeting type="hello" value="hi"/>

Example JSON Accept and Content-Type Headers

> GET /greeting/hello?name=jim
> Accept: application/json,*/*

< 200/OK
< Content-Type: application/json
{ "msg" : "hi, jim" }

Links

RESTful applications dynamically express their state using hyperlinks
- RMM Level 3 characteristic use of links
REST-like APIs do not include that level of complexity
if links are used — use likely constrained to standard response headers

Example Response Headers with Links

POST http://localhost:8080/ejavaTodos/api/todo_lists
{"name":"My First List"}
=> Created/201
Location: http://localhost:8080/ejavaTodos/api/todo_lists/My%20First%20List (1)
Content-Location: http://localhost:8080/ejavaTodos/api/todo_lists/My%20First%20List (2)

1	Location expresses the URI to the resource just acted upon
2	Content-Location expresses the URI of the resource represented in the payload

Summary

In this module, we learned that:

technically — terms "REST" and "RESTful" have a specific meaning defined by Roy Fielding
the Richardson Maturity Model (RMM) defines several levels of compliance to RESTFul concepts, with level 3 being RESTful
very few APIs achieve full RMM level 3 RESTful adoption
- but that is OK!!! — there are many useful and powerful WWW constructs easily made available before reaching the RMM level 3
- can be referred to as "REST-like", "HTTP-based", or "RMM level 2"
- marketers of the world attempting to leverage a buzzword, will still call them REST APIs
most serious REST-like APIs adopt
- HTTP
- multiple resources identified through URIs
- HTTP-compliant use of methods and status codes
- method implementations that abide by defined safety and idempotent characteristics
- standard resource representation formats like JSON, XML, etc.