By on October 21, 2021

The Life of an API Gateway Request (Part 2)

In part 1 of this series, we started a journey from the planet-spanning infrastructure layer to what happens inside a single Kong worker, similar to an office building in complexity. In this second part, we’ll dive a bit deeper—we’ll see who the occupants of that office building are and the kind of life they live.

3. Phases

A phase is a period of time in which something interesting happens, and each phase conditions what can or cannot be done on it.

Init Phase

The init  phase happens only once per Kong node, right after Kong is started. During this time, the Kong node isn’t processing requests yet, but the Master Process is initializing a lot of things like:

  • Executing once per Kong node
  • Usually elevating rights
  • Creating Lua Virtual Machine
  • Loading configuration
  • Checking migration status
  • Loading plugins
  • Initializing Database/DBLess/Hybrid
  • Initializing DNS
  • Initializing PDK
  • Initializing Certificates
  • Initializing Router and Plugin Iterator
  • Not being available to workers

The two most important points are in bold above. First, the Lua Virtual Machine is initialized in this phase. The initialization consists of running Kong’s Lua code, further modifying the virtual machine’s memory, adding functions and modules, and setting a global state. Afterward, this initial virtual machine is “frozen,” to be cloned with everything else by the Worker Processes when they are forked.

The second important point is that the init  phase isn’t available to workers because they’re not created yet when it gets executed.

Init_worker Phase

The phase after init is the init_worker phase. This is executed once per worker, right after the worker has been forked from the Master Process. In this phase, the workers are still not processing requests, but each has their own copy of the master Lua Virtual Machine.

All of the workers will have access to what the Master Process set up during the init phase. Any modifications each worker does afterward, including during init_worker, will only be available for that worker. This makes modification/setting of global variables difficult and inadvisable outside of the init phase. Suppose you need to share and modify global information. In that case, there are other means at your disposal, such as OpenResty’s Shared Dictionaries, which it can use to share information inside a Kong node. The Database, if available, can be used to share information across a cluster.

During the init_worker phase, actions that take place include:

  • Executing once per worker
  • Usually, having less privileged rights than init 
  • Executing a copy of the Master Lua VM
  • Initializing random seed
  • Initializing and subscribing cluster events
  • Initializing and subscribing worker events
  • Initializing and warming up cache
  • Initializing load balancer and health checks
  • Initializing anonymous reports
  • Starting external plugin servers
  • Executing init_worker  phase handler on plugins

Pre-Proxy Phases

As a reminder, when we left, our Kong-Worker-Office-Block looked like this:

kong-gateway-processes-worker-threads-timer-processing

In the above diagram, we see a group of connections, each representing a request being treated by the worker thread—some available, some blocked and one of them (or a timer, like in the case above) being currently processed by the single worker thread.

When a request arrives at the Kong node…

kong-gateway-node-request

…it eventually gets routed to one of the workers and to a connection inside that worker:

kong-gateway-connection-worker

If the worker was an office building, we could think of each connection as a person. And processing a request is this person’s day job.

When a request arrives at the Kong Worker, a connection gets assigned, and that connection’s “work day” starts. It ends when the response is sent back to the consumer (not always, as we will see).

Each connection can be in a separate phase. Similar to the init and init_worker phases, the connection’s phase restricts the type of action that can take place.

The first group of phases that connections get into when a request arrives in Kong is “ pre-proxy.” These would be the “starting the day” activities for a human (having coffee, taking a shower, etc.).

Certificate Phase

This phase works with the Keep-Alive connections. When there is a TLS request, a TLS handshake takes place between the consumer and Kong. This operation is somewhat expensive, so the connection that did the handshake is “kept alive” in case more TLS requests arrive, saving a second handshake.

The certificate gets executed when the TLS handshake completes, and it is not executed on Keep-Alive connections.

Since this connection isn’t always executed, the context, which is a variable that leaves for the duration of the request, gets reset after this phase finishes. This means that if you create a plugin that sets something in the context ( ngx.ctx) of the certificate phase, it will not survive until the next phase. 

Rewrite Phase

The next phase is the rewrite phase; it gets executed very early before the Router has been executed. As a result, the connection still does not “know” which request it belongs to. On a human, it would be similar to being half-awake but still not fully without remembering one’s own identity.

Because of the few things we can do in this phase, I tend to think about it as an “early exit” phase. For example, if you are sure that your Kong node will only answer GET requests, you could implement a very simple plugin that rejects any non-GET request in the rewrite phase. This would be a bit more efficient than doing it later since the worker won’t have to execute the Router on those connections. Almost no Kong plugins use this phase.

Access Phase (HTTP)

The access phase, on the other hand, is the most commonly used. This is the “coffee” or the “breakfast”—something that most people rarely skip. After the Router has been executed, the connection already knows which Route and Service are being matched.

Preread Phase (TCP/UDP)

preread is the same or the equivalent of access, but for TCP or UDP connections. The Router is also executed, and you have a preread phase on plugins for those plugins that support TCP and UDP.

Proxy Phases

After all the “pre-proxy” phases are done, it’s time to go to work. For a connection, that means entering the “proxy phases.”

kong-gateway-proxy-phases

That is like commuting to work and then doing work outside your home more or less.

Content Phase

This is the phase in which Kong sends the request to the Ustream Service and gets a response. It’s implemented as a C module. As a result, it is not available to Lua (and therefore not available to plugins).

Balancer Pseudo-Phase

balancer is a “pseudo-phase” that happens as part of the content phase. balancer  can happen several times per connection: If several upstream servers are configured to match the current request, the first one will be contacted. If it doesn’t answer, then the second one will be contacted. And so on until one of them answers, or all of them fail.

Each time an upstream server fails to connect, the worker thread:

  • Reports passive health check status
  • Executes the Balancer
  • Sets Host or Authority header
  • Handles Keep-Alive with the Upstream
  • Sets Target

We call balancer a “pseudo-phase” because it happens inside content. We cannot, for now, make it available for plugins.

Response Phases

The response phases start as soon as the first bits from the Upstream server response arrive in Kong. The response is treated in chunks. As soon as each chunk is treated, it gets transmitted to the Consumer.

kong-gateway-response-phases

header_filter Phase (HTTP)

The first response, “chunk,” is the one containing the headers. The header_filter phase starts when the last byte from the response headers arrives. At the end of this phase, Kong will send the headers back to the consumer.

This means that the consumer can start getting the headers while Kong is still processing the response body. This is the last opportunity plugins have to modify or add response headers.

body_filter Phase (HTTP)

The body_filter phase concerns itself with the response body. But contrary to the header_filter phase, the response body can be further divided into multiple chunks; as soon as each chunk arrives at Kong, it can be treated and sent back to the Consumer.

This happens because responses are typically bigger than requests or headers; Images, videos and compressed files are all sent inside the response body. By treating them in chunks, Kong avoids having to store them in memory or disk.

A good metaphor is that the response is a chocolate bar, and each connection consumes them “square by square” instead of eating the whole bar in one go.

chocolate

Splitting the response in chunks is very efficient but also a bit uncomfortable. Even though modifying JSON responses is a very common request, it’s awkward, and other operations, like adding a response header depending on a JSON body field, are impossible. 

Response Pseudo-Phase (HTTP)

When we started developing non-Lua plugins (JavaScript, Go, Python), we realized we needed a new pseudo-response that allowed skipping the “chunky treatment” of responses. The new response pseudo-phase groups the response headers and response body chunks into a single step. JSON parsing of the response body is trivial, but now we have to keep in mind that some responses can take a lot of memory. Don’t use this pseudo-phase if you are serving videos.

When a plugin has a response handler function, the header-filter and body-filter get replaced by it for the connection that is treating a request. In the video above, that part is incorrectly explained—the replacing is not global.

Log Phase

log phase is interesting because it happens after the response has been sent back to the Consumer.

kong-gateway-log-phase

It’s sometimes considered “free time” because the worker will not impact that particular connection’s response time. However, all the time the worker thread is “busy” during the log phase is the time it does not spend serving other connections or timers, so it still impacts the global efficiency of the system. 

A usual pattern is communicating with the outside world during the log phase (for example, for logging purposes). OpenResty usually provides cosockets (a non-blocking alternative to OS sockets) in other phases to do this, but they are not available in log. A workaround for this limitation is using a timer—timers can use cosockets to send the information instead. 

Beware, though: Setting one timer per request received is an anti-pattern that has given us trouble in the past. Instead, consider using a single periodic timer that empties a buffer.

Continuing with our connection-person analogy, log would be similar to dreaming at night, which we humans do after work.

dog-sleeping

Handle_error Pseudo-Phase (HTTP)

This pseudo-phase runs anytime something unexpected happens: an unmatched Route, the upstream not responding, etc. When this pseudo-phase gets executed, the header-filter , body-filter  and log  phases happen with a default, non-user-configurable response. It’s not available for plugins, partially because a plugin’s code itself can cause errors.

API Gateway Connection Phases Schema

If you wanted to see this on the schema altogether, it looks like this.

API Gateway Connection Phases Schema

4. Plugins

Plugins, the last layer of our abstraction tour, are pieces of code inside Kong that perform actions.

If connections were people, then plugins would be their hands.

It turns out that plugins can only do four types of action.

Plugins can talk with an external system, and this is what our logging and tracing plugins, like Zipkin and Prometheus, do. 

kong-gateway-log-phase-diagram

The ideal place for doing it is the log phase (with the note above about cosockets, timers and antipatterns). 

Plugins can also modify the two arrows that exit Kong on the diagram above. Some plugins, like request-transformer and correlation-id, can modify the request before sending it upstream during the access phase.

kong-gateway-access-phase-diagram

Other plugins, like response-transformer  and cors, act on the other arrow that exists in Kong, modifying the upstream response before sending it to the Consumer.

kong-gateway-access-phase-plugins

cors is unusual in that it sets its response headers on the access phase. In general, header-filter and body-filter are the usual places to modify the response. 

A specialization of sending a custom response is shortcutting. This occurs when Kong decides to cancel the proxying altogether and instead responds based on the request.

All of the authorization plugins shortcut requests if their credentials are incorrect or non-existing. Others, like bot-detection or IP-restriction, shortcut undesired traffic, protecting the upstream server from overloads. Shortcutting usually occurs in the access phase because the Router has already been executed, but the request has not left Kong yet.

kong-gateway-access-phase-diagram-plugins

Conclusion

It turns out that a lot of the effective work that the Kong Worker people do occurs either during breakfast, before leaving for work or while dreaming at night. And they prefer to eat their chocolate square by square.

Have questions or want to stay in touch with the Kong community? Join us wherever you hang out:

Star us on GitHub

🐦 Follow us on Twitter

🌎 Join the Kong Community

🍻 Join our Meetups

❓ ️Ask and answer questions on Kong Nation

💯 Apply to become a Kong Champion

Tags: