Build a pipe from container to cluster

There are valid needs to talk to the Kubernetes cluster from segregated docker containers. It’s possible to do so:

Build the pipe from the container to the host machine

There are several ways to connect the host machine. the container is running together with the host, behaving like on the same subnet. you can access it through the public IP.

otherwise, more elegantly, you can leverage on host.docker.internal to talk to the host

Proxy the resources for the Kubernetes cluster

you can start a Kubernetes proxy to talk to the cluster.

kubectl proxy --port=8080 --disable-filter &

Access to the resources

then to talk to the resources in the cluster from the container, you can do

host.docker.internal:8080/api/v1/namespaces/default/services/pod:{port}/proxy/

This is related to, https://lwpro2.dev/2020/04/03/expose-existing-deployment-with-minikube/, as there is an issue with Kubernetes apiserver proxy, https://github.com/kubernetes/kubernetes/issues/89360, which would strip out the parameter for websocket.

The work around is to use minikube tunnel with LoadBalancer, as in above post.

openssl issue on OSX

in addition to, https://lwpro2.dev/2020/04/20/git-command-issue-with-openssl/, even though `brew uninstall, brew install` the specific old version of openssl worked for me with git.

however, it still breaks with poetry.

[SSLError]
HTTPSConnectionPool(host='privatepypi', port=443): Max retries exceeded with url: /pypi/table-understanding/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not availa
ble."))

the solution to work out for both is

brew switch openssl 1.0.2s

private pypi server issue

in addition to, https://lwpro2.dev/2020/04/20/build-a-private-pypi-with-github/, it has took me quite a while to figure out why certain packages i am not able to install using poetry:

Turns out, that’s because PEP has a very weird renaming:

https://www.python.org/dev/peps/pep-0503/#normalized-names

hence, i need to publish my package named as safe-logging, even though the binary is safe_logging:

even though, within the directory, it is named safe_logging

leverage on eslint and prettier together

eslint is now the defector linter for front end, even for typescript, especially with the deprecation of tslint.

eslint is able to do both formatting, like line length, trailing semi colon etc; at the same time, it’s able to do syntax checking as well, for example, unused or undefined variables alike.

while at the same time, prettier is really doing a good job for formatting the front end. it’s a very opinionated framework however, with very easy to customize configurations. for example,

{  
"semi": true,  
"trailingComma": "all",  
"singleQuote": true,  
"printWidth": 70
}

with the combination of both, we can leverage on the strength of both, to do lint and format:

npx eslint -c .eslintrc.json **/*.{ts,tsx} --fix ##for format or fix

npx eslint -c .eslintrc.json **/*.{ts,tsx} ## for lint

these are the set ups we need to prepare:

install the dependencies:

npm install -g prettier eslint
## or
yarn add prettier eslint

install the plugins

npm install --save-dev eslint-config-prettier eslint-plugin-prettier
## or
yarn add --dev eslint-config-prettier eslint-plugin-prettier

then configure .eslintrc

{
"root": true,
"parser": "@typescript-eslint/parser",
"plugins": ["@typescript-eslint","prettier"],
"extends": [
"eslint:recommended",
"plugin:@typescript-eslint/eslint-recommended",
"plugin:@typescript-eslint/recommended",
"plugin:prettier/recommended"
],
"rules": {
"@typescript-eslint/interface-name-prefix": 1
}
}


in addition, to use airbnb formatting:

npx install-peerdeps --dev eslint-config-airbnb

then add it into .eslintrc.json

{  
"extends": ["airbnb", "prettier"],  
"plugins": ["prettier"],  
"rules": {    
"prettier/prettier": ["error"]  
}}

Build a private pypi with github

recently, I have tried to build a pypi server on top of github.

here are the steps:

  1. create a repo, to host those packages
  2. at the root level, define the setup.py with the findpackages(), like
import setuptools

with open("README.md", "r") as fh:
   long_description = fh.read()

setuptools.setup(
   name="pypi",
   version="0.0.1",
   author="lwpro2",
   author_email="lwpro2",
   description="A pypi for python packages",
   long_description=long_description,
   long_description_content_type="text/markdown",
   url="https://github.com/lwpro2",
   packages=setuptools.find_packages(),
   classifiers=[
      "Programming Language :: Python :: 3",
      "License :: OSI Approved :: MIT License",
      "Operating System :: OS Independent",
   ]
)

3. at each packages, add the setup.py as well

import setuptools

with open("README.md", "r") as fh:
   long_description = fh.read()

setuptools.setup(
   name="package1",
   version="0.0.2",
   author="lwpro2",
   author_email="lwpro2",
   description="A fake library",
   long_description=long_description,
   long_description_content_type="text/markdown",
   url="https://github.com/lwpro2",
   packages=setuptools.find_packages(),
   classifiers=[
      "Programming Language :: Python :: 3",
      "License :: OSI Approved :: MIT License",
      "Operating System :: OS Independent",
   ]
)

the folder structure will be like

--pypi
------setup.py
------README.md
------package1
-------------setup.py
-------------README.md
-------------__init__.py
-------------functions.py
------package2
-------------setup.py
-------------README.md
-------------__init__.py
-------------functions.py

4. run this command to generate the binaries

python3 setup.py sdist bdist_wheel

the folder will become

--pypi
------setup.py
------README.md
------package1
-------------setup.py
-------------README.md
-------------__init__.py
-------------functions.py

-------------build/
-------------dist/
--------------------package1-0.0.1.tar.gz
------package2
-------------setup.py
-------------README.md
-------------__init__.py
-------------functions.py

-------------build/
-------------dist/
--------------------package2-0.0.1.tar.gz

5. then create a static pypi server which comply to Pep 503, like



6. then add the new pypi server into Pipfile and pyproject.toml

Pipfile

[source]
name = "privatepypi"
url = "http://private/pypi"
verify_ssl = false

[dev-packages]

[packages]
package1 = {version = "*", index = "privatepypi"}

pyproject.toml

[[tool.poetry.source]]
name = "privatepypi"
url = "https://private/pypi"
secondary = true

[tool.poetry.dependencies]
package2= "^0.0.1"

7. then can install packages using the commands as normal

pip install

poetry install

Possible Kubernetes exception

in addition to https://lwpro2.dev/2020/04/03/expose-existing-deployment-with-minikube/, with kubectl proxy --port=8080 --disable-filter, the container could talk to the kubernetes cluster.

however, if the proxy is not running or not running properly, it could result in two types of exceptions:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.8/http/client.py", line 1230, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1276, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1225, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1004, in _send_output
self.send(msg)
File "/usr/local/lib/python3.8/http/client.py", line 944, in send
self.connect()
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 184, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 168, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused

alternatively,

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 13463, in list_namespaced_service
(data) = self.list_namespaced_service_with_http_info(namespace, **kwargs) # noqa: E501
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 13551, in list_namespaced_service_with_http_info
return self.api_client.call_api(
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 340, in call_api
return self.__call_api(resource_path, method,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 172, in __call_api
response_data = self.request(
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 362, in request
return self.rest_client.GET(url,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 237, in GET
return self.request("GET", url,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (502)
Reason: Bad Gateway
HTTP response headers: HTTPHeaderDict({'Date': 'Tue, 14 Apr 2020 06:03:12 GMT', 'Content-Length': '0'})