README.md · master · KN / tyk-api

TT-13269 - Refactor/streams (#6593) · aee8137b
Tit Petric authored Oct 19, 2024
### **User description**
Extracts streaming middleware into internal/middleware/stream;
Implements a number of interfaces to cover gateway/base middleware
Implements `wrappedMiddleware` to allow middlewares with less coupling.

I tried to update couplings along as I went, it's passing `go build` and
`go test -c`.


___

### **PR Type**
enhancement, tests


___

### **Description**
- Refactored streaming middleware by introducing a new `Manager` for
handling streams and updated the `StreamingMiddleware` to use this new
structure.
- Replaced various `apidef` types with `model` types across multiple
files to improve modularity and reduce coupling.
- Introduced `WrapMiddleware` functionality to allow middleware wrapping
with less coupling.
- Updated test cases to align with the new streaming middleware
implementation and the use of the `model` package.
- Added new interfaces in the `model` package to define common behaviors
for Gateway and Middleware components.
- Updated method calls to use the new `ReplaceTykVariables` function for
consistency.


___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>22
files</summary><table>
<tr>
  <td>
    <details>
<summary><strong>api_loader.go</strong><dd><code>Integrate new streaming
middleware and wrap functionality</code></dd></summary>
<hr>

gateway/api_loader.go

<li>Added import for <code>internal/middleware/stream</code>.<br> <li>
Replaced <code>StreamingMiddleware</code> with a new streaming
middleware <br>implementation.<br> <li> Introduced
<code>WrapMiddleware</code> for middleware wrapping.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-cdf0b7f176c9d18e1a314b78ddefc2cb3a94b3de66f1f360174692c915734c68">+9/-4</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>gateway.go</strong><dd><code>Update Gateway interface
implementation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></summary>
<hr>

gateway/gateway.go

- Updated interface implementation to use `model.Gateway`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-17cb8b37eda9018fe1c6cdb5f96b3fc948fc8ba49bc516987b8269576db9fcd4">+2/-7</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>health_check.go</strong><dd><code>Use model package for
health check items</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></summary>
<hr>

gateway/health_check.go

- Replaced `apidef.HealthCheckItem` with `model.HealthCheckItem`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-978a2d1427d9209765e541618af10683944c6396df1a6fb8b5221e4f16658a6a">+31/-30</a>&nbsp;
</td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>middleware.go</strong><dd><code>Update TykMiddleware
interface and constants</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

gateway/middleware.go

<li>Added <code>GetSpec</code> method to <code>TykMiddleware</code>
interface.<br> <li> Changed constant <code>mwStatusRespond</code> to use
<code>middleware.StatusRespond</code>.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-703054910891a4db633eca0f42ed779d6b4fa75cd9b3aa4c503e681364201c1b">+10/-12</a>&nbsp;
</td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>middleware_wrap.go</strong><dd><code>Introduce
middleware wrapping functionality</code>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></summary>
<hr>

gateway/middleware_wrap.go

<li>Introduced <code>wrapMiddleware</code> struct for middleware
wrapping.<br> <li> Implemented <code>WrapMiddleware</code> function.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-1da43bd02220acad12bce8d8c5600b4acfee3c40f90c53825802747004c9fb0a">+51/-0</a>&nbsp;
&nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>mw_auth_key.go</strong><dd><code>Update variable
replacement method call</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></summary>
<hr>

gateway/mw_auth_key.go

- Updated method call to `ReplaceTykVariables`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-aeba053023a54c723dd9f83837e29ca0b2d9a212bc98fa6ad4bbb062669a1cf0">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>mw_graphql.go</strong><dd><code>Update variable
replacement method call</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></summary>
<hr>

gateway/mw_graphql.go

- Updated method call to `ReplaceTykVariables`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-c46e0f07348c8f519e5912f4394f048f43c1e3fb5063c27245272c8f645b4cab">+2/-2</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>mw_modify_headers.go</strong><dd><code>Update variable
replacement method call</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></summary>
<hr>

gateway/mw_modify_headers.go

- Updated method call to `ReplaceTykVariables`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-061b7bd9b76abe91c660494762ad868c6cc6135fdf1e97465377df39e1eeac8e">+2/-2</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>

<summary><strong>mw_persist_graphql_operation.go</strong><dd><code>Update
variable replacement method call</code>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

gateway/mw_persist_graphql_operation.go

- Updated method call to `ReplaceTykVariables`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-ace7a721ebc7c7d61ce5a9f01b906cfe617aedf48364f08f40ccfde670685ca6">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>mw_rate_limiting.go</strong><dd><code>Update variable
replacement method call</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></summary>
<hr>

gateway/mw_rate_limiting.go

- Updated method call to `ReplaceTykVariables`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-4bf8ae01ccab67bb786468f793f6bb4324c8f6b950b0e98e203effebe763a630">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>mw_transform.go</strong><dd><code>Update variable
replacement method call</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></summary>
<hr>

gateway/mw_transform.go

- Updated method call to `ReplaceTykVariables`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-d7a3cdc3dcabd415dffee6c044ea27dbe877add0ddc42471e10943125693fc12">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>mw_url_rewrite.go</strong><dd><code>Rename and document
ReplaceTykVariables function</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

gateway/mw_url_rewrite.go

<li>Renamed <code>replaceTykVariables</code> to
<code>ReplaceTykVariables</code> and added <br>documentation.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-84a6a5c810334aaa8702669f2aebf0284f116d83e8a55ec9d1d5b8bae87f1be6">+6/-2</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>

<summary><strong>res_handler_header_injector.go</strong><dd><code>Update
variable replacement method call</code>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

gateway/res_handler_header_injector.go

- Updated method call to `ReplaceTykVariables`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-db30365a54a06d3fbf9f4aad3fe133de85dbb75fae2177d84ede3214407f31ca">+3/-3</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>rpc_storage_handler.go</strong><dd><code>Use model
package for RPC storage handler types</code>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

gateway/rpc_storage_handler.go

<li>Replaced <code>apidef</code> types with <code>model</code> types for
RPC storage handling.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-8875f75b602664c44b62b67a4da41d748124ad270573a44db4ec977ee5d68021">+24/-25</a>&nbsp;
</td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>const.go</strong><dd><code>Add StatusRespond constant
for middleware</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></summary>
<hr>

internal/middleware/const.go

- Added constant `StatusRespond` for middleware processing.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-8a6d0644eb989a86bc580dcdcee0ac94316c7883c7516d3e11e55c66a833965a">+5/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>stream_manager.go</strong><dd><code>Introduce Manager
for stream management</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></summary>
<hr>

internal/middleware/stream/stream_manager.go

<li>Introduced <code>Manager</code> struct for stream management.<br>
<li> Implemented stream creation and removal functionalities.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-ed75e572397b02c39d91dcca90c9bfd84f527fa4b5522c0a08f7ee9e44585ef5">+133/-0</a>&nbsp;
</td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>streaming_middleware.go</strong><dd><code>Refactor
StreamingMiddleware with new Manager</code>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

internal/middleware/stream/streaming_middleware.go

<li>Refactored <code>StreamingMiddleware</code> to use new
<code>Manager</code>.<br> <li> Updated logging and stream management
logic.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-99c21a93ded6b6c0fcfba1af1b9a2189df150da067ecdb08cf33b1aced9242e3">+62/-133</a></td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>health_check.go</strong><dd><code>Rename package to
model</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

internal/model/health_check.go

- Renamed package from `apidef` to `model`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-44e3299d864d891747443f0c999d95e7d19410b67817fdfd1c5840c616fdcd64">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>interfaces.go</strong><dd><code>Add interfaces for
Gateway and Middleware</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></summary>
<hr>

internal/model/interfaces.go

- Introduced interfaces for Gateway and Middleware.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-43ba6dd4a8d193850dea32e8af5c361470cd62bfa390c580a39f7142a56bd391">+66/-0</a>&nbsp;
&nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>rpc.go</strong><dd><code>Move RPC types to model
package</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

internal/model/rpc.go

- Moved RPC-related types to `model` package.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-a9cdd23914e87a8671ec07c0a83257f8bbd12dc8939c5014cf368b796bf8ade4">+2/-6</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>apply.go</strong><dd><code>Use model.PolicyProvider in
Service struct</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

internal/policy/apply.go

- Updated `Service` struct to use `model.PolicyProvider`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-59b92e9d31f142f1d99b746eb3ff7db4e26bf6c3044c9b87b58034a947ee04d1">+3/-10</a>&nbsp;
&nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>synchronization_forcer.go</strong><dd><code>Use model
package for GroupLoginRequest</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

rpc/synchronization_forcer.go

- Replaced `apidef.GroupLoginRequest` with `model.GroupLoginRequest`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-97417011065a292f63eeb6fb031afbcfffa75cb3fc7073f8431add277b250c98">+2/-2</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

</table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>3
files</summary><table>
<tr>
  <td>
    <details>
<summary><strong>mw_streaming_test.go</strong><dd><code>Update streaming
test cases to use new package</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

gateway/mw_streaming_test.go

<li>Updated test cases to use <code>stream</code> package for streaming
<br>functionalities.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-a0d1bd0196a741537a3c850e340225c8993e49d709c838af0f1b48b9893af1da">+8/-7</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>rpc_storage_handler_test.go</strong><dd><code>Update
RPC storage handler tests to use model package</code>&nbsp; &nbsp;
&nbsp; &nbsp; </dd></summary>
<hr>

gateway/rpc_storage_handler_test.go

- Updated tests to use `model` package for RPC storage handler.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-69de989a02b3bc32ae376c514ee84633c609200db22385c0e16c361d6ea74cd6">+20/-23</a>&nbsp;
</td>

</tr>                    

<tr>
  <td>
    <details>
<summary><strong>rpc_test.go</strong><dd><code>Update RPC tests to use
model package</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

gateway/rpc_test.go

- Updated RPC test cases to use `model` package for API definitions.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6593/files#diff-68d6d05f22702a24741c6e233a2cb1f227dacf0309d7d94f651038ac7b90f49e">+5/-5</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>                    
</table></details></td></tr></tr></tbody></table>

___

> 💡

 **PR-Agent usage**: Comment `/help "your question"` on any pull
request to receive relevant information

---------

Co-authored-by: Tit Petric <tit@tyk.io>
Co-authored-by: Leonid Bugaev <leonsbox@gmail.com>
aee8137b