the default `gradlew bootRun` command is not always working out. The latest error message I got is
as an alternative, the plain build works, even though with a caveat:
caveat: the built out package has been put under libs folder, instead of what the bootScripts is expecting (lib). So point to the right path will work.
PS C:\Users\...> .\build\bootScripts\cli.bat
Error: Unable to access jarfile C:\Users\...\build\bootScripts\..\lib\cli-0.0.1-SNAPSHOT.jar
PS C:\Users\...\build> java -jar .\libs\cli-0.0.1-SNAPSHOT.jar
2020-08-23 10:08:55.583 INFO 53436 --- [ main] : Starting Application on ....
2020-08-23 10:08:56.493 INFO 53436 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2020-08-23 10:08:56.580 INFO 53436 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 72ms. Found 3 JPA repository interfaces.
2020-08-23 10:08:57.191 INFO 53436 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-08-23 10:08:57.823 INFO 53436 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2020-08-23 10:08:57.864 INFO 53436 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
it’s not unusual that there are cases we have new graphql servers to be added, or existing graphql servers updated.
Apollo server at the moment, have a gateway component which could statically maintain and route the right traffic to the right server and aggregate results.
However, this community version of Apollo server and gateway is not able to cater for the case in the beginning, where we would like to hot update/reload the gateway with any updates however with shut down the servers.
After some trials and errors, this is the work around to sort this out:
instead of hardcode the service list (apollo graphql servers), maintain it separately and dynamically:
#### instead of this
const gateway = new ApolloGateway({
serviceList: [
{ name: "astronauts", url: "http://localhost:4002" },
{ name: "missions", url: "http://localhost:4003" }
]
});
## maintain this list
#### instead of this
const dynamicServiceList = [
{ name: "astronauts", url: "http://localhost:4002" },
{ name: "missions", url: "http://localhost:4003" }
];
then from apollo gateway, instead of spoon feeding the list, switch it to dynamically load the definitions
const gateway = new ApolloGateway({
serviceList: [],
...
experimental_updateServiceDefinitions: async (serviceList) => {
return {
isNewSchema: true,
serviceDefinitions: await Promise.all(dynamicServiceList.map(async (service) => {
//load the sql
const data = await request(service.url, sdlGQL);
...
return {
...
//then feed the data here
name: service.name,
url: service.url,
typeDefs: parse(data._service.sdl)
};
}))
};
}
});
At the same time, create a new endpoint, if needed, to take in updated servers, or new servers, so that to update the dynamic list
app.get('/hot_reload_schema/', async (req, res) => {
//get the new server info from req
....
const status = await validateSDL(name, url);
if (status){
//update the dynamic list
dynamicServiceList.push({name, url});
res.send('The new schema has been successfully registered');
}
else {
res.send('Please check the log for the error details while registering this schema');
}
}
);
lastly, make sure the gateway is polling the service list at some intervals:
new ApolloGateway({
pollingTimer: 100_000,
experimental_pollInterval: 100_000,
....
});
alternatively, this could also be updated by an explicit load:
new ApolloGateway({
....
}).load();
have contributed the solution to the community here.
lua-resty-openidc is a certified OIDC and OAuth library built onto openresty. While openresty is a reverse proxy built on nginx with lua and luaJit embedded, which greatly upgrade nginx’s capability.
lua-resty-openidc is able to authenticate and authorize the client with compliant OP (keycloak in my case). However, I was facing issues with infinite redirects:
location /test {
access_by_lua_block {
local opts = {
discovery = "http://keycloak/...../.well-known/openid-configuration",
redirect_uri_path = "/test",
accept_none_alg = true,
client_id = "xxxx",
client_secret = "xxxxx",
use_nonce = true,
revoke_tokens_on_logout = true,
}
local res, err, url, session = require("resty.openidc").authenticate(opts)
if err or not res then
ngx.status = 403
ngx.say(err and err or "no access_token provided")
ngx.exit(ngx.HTTP_FORBIDDEN)
end
}
default_type text/html;
content_by_lua 'ngx.say("<p>hello, world here from test</p>")';
}
for above block, I was expecting the library able to direct the client to keycloak authentication at first time, then subsequently redirect back to the redirect_uri /test, which then see the client is already authenticated, and proceed to the content_by_lua` block.
a follow up to the original post, the redirect_uri itself could be causing the issue. Instead of pointing it to a final landing page, point it to a intermittent place which would then be directed to the original place (the protected location) should sort the problem as well.
I was facing some issue when embedding the code into google site:
turns out google was possibly running some checks, maybe the traffic while the code is added, and with some activities from any add-on or chrome itself, google site think it’s suspicious and blocked it.
the solution was to run chrome in incongnito mode, it will work fine.
have been facing various different issues with one EKS cluster,
the pods is stuck at failedMount, even though the PV and PVC has been in bound state, and the PVC also shows bound to the specific pods, the EFS on aws is also in good state
## relevant error message
list of unattached/unmounted volumes
Warning FailedMount 1m (x3 over 6m) kubelet, ..... Unable to mount volumes for pod "mypod1_test(c038c571-00ca-11e8-b696-0ee5f3530ee0)": timeout expired waiting for volumes to attach/mount for pod "test"/"mypod1". list of unattached/unmounted volumes=[aws]
Normal Scheduled 5m45s default-scheduler Successfully assigned /test3 to ....
Warning FailedMount 83s (x2 over 3m42s) kubelet, ... Unable to mount volumes for pod "test3_....(fe03d56e-aa26-11ea-9d9c-067c9e734f0a)": timeout expired waiting for volumes to attach or mount for pod ""/"test3". list of unmounted volumes=[mypd]. list of unattached volumes=[mypd default-token-mznhh]
related, it shows the efs csi driver not available
aws ebs csi driver
Warning FailedMount 9m1s (x4 over 15m) kubelet, ....Unable to mount volumes for pod "test3_...(f0d65986-aa27-11ea-a7b2-022dd8ed078a)": timeout expired waiting for volumes to attach or mount for pod ""/"test3". list of unmounted volumes=[mypd]. list of unattached volumes=[mypd default-token-mznhh]
Warning FailedMount 2m35s (x6 over 2m51s) kubelet, ... MountVolume.MountDevice failed for volume "scratch-pv" : driver name efs.csi.aws.com not found in the list of registered CSI drivers
Warning FailedMount 2m19s kubelet, .... MountVolume.SetUp failed for volume ".." : rpc error: code = Internal desc = Could not mount "fs-43b99802:/" at "/var/lib/kubelet/pods/f0d65986-aa27-11ea-a7b2-022dd8ed078a/volumes/kubernetes.io~csi/scratch-pv/mount": mount failed: exit status 1
Mounting command: mount
Mounting arguments: -t efs fs-43b99802:/ /var/lib/kubelet/pods/f0d65986-aa27-11ea-a7b2-022dd8ed078a/volumes/kubernetes.io~csi/scratch-pv/mount
Output: Failed to resolve "fs-43b99802....amazonaws.com" - check that your file system ID is correct.
See https://docs.aws.amazon.com/console/efs/mount-dns-name for more detail.
pods stuck at termination state, even thought the svc, deployment, and rs has been killed, the pod still has been stuck at termination
the final solution to sort out all is to reboot the EKS worker node
Have been continuous receiving this error message during the `helm upgrade` even with several retry
UPGRADE FAILED
Error: kind StorageClass with the name "..." already exists in the cluster and wasn't defined in the previous release. Before upgrading, please either delete the resource from the cluster or remove it from the chart
Error: UPGRADE FAILED: kind StorageClass with the name ".." already exists in the cluster and wasn't defined in the previous release. Before upgrading, please either delete the resource from the cluster or remove it from the chart
Have tried two solutions, (the 1st doesn’t work out)
as the helm upgrade error message, I did a helm delete the storageclass, then run the helm upgrade, even though on the internet some claim this work, but actually no, it fails again with same error message
did a helm rollback to a previous working release, then run the helm upgrade again, then worked
Microsoft Edge/Opera/Google Chrome has become the de-facto browser for internet. This suite here will greatly further enhance your browsing experience:
Tab & URL manager
Problem statement:
we tend to have so many tabs open, it becomes difficult to find the specific tabs among so many we have opened.
Instead, we normally tend to just open a new tab which has already been open and hidden somewhere, this will additionally consume more of our computer memory usage.
We have visited certain pages before, but really cannot recall the complete address of the page.
We are trying to search for a term, however, would like to see only the top suggestions.
We might have different search engines would like to leverage on, for example maybe bing for desktop images, or baidu for chinese languages.
We have a limited screen, would like to maximize the space screen, instead of leaving the address and tab to consume the precious screen space.
* type a URL followed by enter to visit the specified address
* type a term followed by enter to search the term (the search engine is configurable)
* type a term to navigate to any popular search suggestions
* type a keyword to navigate to any mapped URL (the map is configurable)
* type a term to navigate to any open tabs
* type a term to get the likely result you would like to visit
ShortCut for URL mapper
Problem Statement:
It’s always difficult to remember the long URLs. Yet, we could have so many pages we normally visit.
We all have list of often visited pages each. Normally we would either bookmark them, then click one by one to open each, or just open a new tab each and type the URLs manually.
Recently, I have set up my micro services application onto datadog APM with tracing and logging.
something like,
so that we can follow through, for example, any http request or user_id from HTTP request, controller, service, authentication, database, and etc, with the same tracing ID and logs detailed.
Here are the steps I have followed:
Set up datadog account and access, so that we will get the datadog API key, which is to be used to upload the data to datadog server
install the datadog agent onto the server, we are using kubernetes on AWS (EKS) here, and have installed it as a daemonset (so that we have one for each node) with terraform.
set up the necessary environment variable in the daemonset
so that these logs would be in JSON format, and datadog would be able to parse the logs into tags & attributes.
4. then configure the datadog tracing
from ddtrace import config, patch_all, tracer
def setup_datadog():
tracer.configure(
hostname=os.environ["DD_AGENT_HOST"],#set up in the deployment.yaml, the env attribute
)
tracer.set_tags({"env": "production"})
config.flask["service_name"] = "checkout-service" #this could also set up in the deployment.yaml, the env attribute
config.flask["analytics_enabled"] = True
config.flask["extra_error_codes"] = [401, 403]
patch_all()
5. finally, then annotate the entry point function, with
Sometimes, PV and PVC could stuck at terminating. This is because there is a finalizer to protect the PV and PVC termination while there are still possible usage.
For example, when I am trying to delete the PV
both PV and PVC however are stuck for quite a while
the resolution is to edit and remove the finalizers:
then without the finalizers, it will let the PV and PVC termination.
endpoints is normally used behind the scene, without the need of any manual intervention.
However, for local testing, it could become helpful to leverage on the endpoints customization.
---
kind: "Service"
apiVersion: "v1"
metadata:
name: "svc-to-external-web"
spec:
ports:
- name: "apache"
protocol: "TCP"
port: 80
targetPort: 80
---
kind: "Endpoints"
apiVersion: "v1"
metadata:
name: "svc-to-external-web"
subsets:
- addresses:
- ip: "8.8.8.8" #The IP Address of the external web server
ports:
- port: 80
name: "apache"
for example, with above, if we create a service without any selector (of any pods/deployments), and manually create an endpoint with the same name as the service, then we can configure where the endpoint point to, which could be an external IP address, or maybe to a docker service
Today we’re excited to announce something special for Babel users.Over a year ago, we set out to find what the biggest difficulties users were running into with TypeScript, and we found that a common theme among Babel users was that trying to get TypeScript set up was just too hard. The reasons often varied, but for a lot of developers, rewiring a build that’s already working can be a daunting task.
Babel is a fantastic tool with a vibrant ecosystem that serves millions of developers by transforming the latest JavaScript features to older runtimes and browsers; but it doesn’t do type-checking, which our team believes can bring that experience to another level. While TypeScript itself can do both, we wanted to make it easier to get that experience without forcing users to switch from Babel.
That’s why over the past year we’ve collaborated with the Babel team, and today we’re happy to jointly announce that Babel 7 now ships with TypeScript support!
How do I use it?
If you’re already using Babel and you’ve never tried TypeScript, now’s your chance because it’s easier than ever. At a minimum, you’ll need to install the TypeScript plugin.
npm install --save-dev @babel/preset-typescript
Though you’ll also probably want to get the other ECMAScript features that TypeScript supports:
Using the TypeScript compiler is still the preferred way to build TypeScript. While Babel can take over compiling/transpiling – doing things like erasing your types and rewriting the newest ECMAScript features to work in older runtimes – it doesn’t have type-checking built in, and still requires using TypeScript to accomplish that. So even if Babel builds successfully, you might need to check in with TypeScript to catch type errors. For that reason, we feel tsc and the tools around the compiler pipeline will still give the most integrated and consistent experience for most projects.
So if you’re already using TypeScript, maybe this doesn’t change much for you. But if you’re already using Babel, or interested in the Babel ecosystem, and you want to get the benefits of TypeScript like catching typos, error checking, and the editing experiences you might’ve seen in the likes of Visual Studio and Visual Studio Code, this is for you!
Caveats
As we mentioned above, the first thing users should be aware of is that Babel won’t perform type-checking on TypeScript code; it will only be transforming your code, and it will compile regardless of whether type errors are present. While that means Babel is free from doing things like reading .d.ts files and ensuring your types are compatible, presumably you’ll want some tool to do that, and so you’ll still need TypeScript. This can be done as a separate tsc --watch task in the background, or it can be part of a lint/CI step in your build. Luckily, with the right editor support, you’ll be able to spot most errors before you even save.
Second, there are certain constructs that don’t currently compile in Babel 7. Specifically,
namespaces
bracket style type-assertion/cast syntax regardless of when JSX is enabled (i.e. writing <Foo>x won’t work even in .ts files if JSX support is turned on, but you can instead write x as Foo).
enums that span multiple declarations (i.e. enum merging)
These omissions are largely based technical constraints in Babel’s single-file emit architecture. We believe that most users will find this experience to be totally acceptable. To make sure that TypeScript can call out some of these omissions, you should ensure that TypeScript uses the --isolatedModules flag.
What next?
You can read up on the details from the Babel side on their release blog post. We’re happy that we’ve had the chance to collaborate with folks on the Babel team like Henry Zhu, Andrew Levine, Logan Smyth, Daniel Tschinder, James Henry, Diogo Franco, Ivan Babak, Nicolò Ribaudo, Brian Ng, and Vladimir Kurchatkin. We even had the opportunity to speed up Babylon, Babel’s parser, and helped align with James Henry’s work on typescript-eslint-parser which now powers Prettier’s TypeScript support. If we missed you, we’re sorry but we’re grateful and we appreciate all the help people collectively put in!
Our team will be contributing to future updates in the TypeScript plugin, and we look forward to bringing a great experience to all TypeScript users. Going forward, we’d love to hear your feedback about this new TypeScript support in Babel, and how we can make it even easier to use. Give us a shout on Twitter at @typescriptlang or in the comments below.
There are valid needs to talk to the Kubernetes cluster from segregated docker containers. It’s possible to do so:
Build the pipe from the container to the host machine
There are several ways to connect the host machine. the container is running together with the host, behaving like on the same subnet. you can access it through the public IP.
otherwise, more elegantly, you can leverage on host.docker.internal to talk to the host
Proxy the resources for the Kubernetes cluster
you can start a Kubernetes proxy to talk to the cluster.
kubectl proxy --port=8080 --disable-filter &
Access to the resources
then to talk to the resources in the cluster from the container, you can do
[SSLError] HTTPSConnectionPool(host='privatepypi', port=443): Max retries exceeded with url: /pypi/table-understanding/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not availa ble."))
eslint is now the defector linter for front end, even for typescript, especially with the deprecation of tslint.
eslint is able to do both formatting, like line length, trailing semi colon etc; at the same time, it’s able to do syntax checking as well, for example, unused or undefined variables alike.
while at the same time, prettier is really doing a good job for formatting the front end. it’s a very opinionated framework however, with very easy to customize configurations. for example,
however, if the proxy is not running or not running properly, it could result in two types of exceptions:
Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 387, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/local/lib/python3.8/http/client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/local/lib/python3.8/http/client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.8/http/client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.8/http/client.py", line 1004, in _send_output self.send(msg) File "/usr/local/lib/python3.8/http/client.py", line 944, in send self.connect() File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 184, in connect conn = self._new_conn() File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 168, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
alternatively,
Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 13463, in list_namespaced_service (data) = self.list_namespaced_service_with_http_info(namespace, **kwargs) # noqa: E501 File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 13551, in list_namespaced_service_with_http_info return self.api_client.call_api( File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 340, in call_api return self.__call_api(resource_path, method, File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 172, in __call_api response_data = self.request( File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 362, in request return self.rest_client.GET(url, File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 237, in GET return self.request("GET", url, File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 231, in request raise ApiException(http_resp=r) kubernetes.client.rest.ApiException: (502) Reason: Bad Gateway HTTP response headers: HTTPHeaderDict({'Date': 'Tue, 14 Apr 2020 06:03:12 GMT', 'Content-Length': '0'})
Both $VARIABLE and ${VARIABLE} syntax are supported. Additionally when using the 2.1 file format, it is possible to provide inline default values using typical shell syntax:
${VARIABLE:-default} evaluates to default if VARIABLE is unset or empty in the environment.
${VARIABLE-default} evaluates to default only if VARIABLE is unset in the environment.
Similarly, the following syntax allows you to specify mandatory variables:
${VARIABLE:?err} exits with an error message containing err if VARIABLE is unset or empty in the environment.
${VARIABLE?err} exits with an error message containing err if VARIABLE is unset in the environment.
There could be need to talk to Kubernetest cluster from segregated docker containers. It’s possible to do so:
build the pipe from the container to the host machine
There are several ways to connect the host machine. the container is running together with the host, behaving like on the same subnet. you can access it through the public IP.
otherwise, more elegantly, you can leverage on host.docker.internal to talk to the host
proxy the resources for the Kubernetes cluster
you can start a Kubernetest proxy to talk to the cluster.
kubectl proxy --port=8080 --disable-filter &
then to talk to the resources in the cluster from the container, you can do
What’s the difference between a Service and a Deployment in Kubernetes?
A deployment is responsible for keeping a set of pods running.
A service is responsible for enabling network access to a set of pods.
We could use a deployment without a service to keep a set of identical pods running in the Kubernetes cluster. The deployment could be scaled up and down and pods could be replicated. Each pod could be accessed individually via direct network requests (rather than abstracting them behind a service), but keeping track of this for a lot of pods is difficult.
We could also use a service without a deployment. We’d need to create each pod individually (rather than “all-at-once” like a deployment). Then our service could route network requests to those pods via selecting them based on their labels.
Services and Deployments are different, but they work together nicely.
With the increased popularity of Graphql, so does the popularity of the frameworks such as (for python) graphene and Ariadne.
They are two frameworks able to reach the same target: a working GraphQL backend service catering for Graphql queries, mutation, and subscriptions.
However, they are taking opposite routes.
Graphene is taking code first route. So developers would write the python to tell the services the information it can server (resolve or mutate) and how to serve them. Something like:
import graphene
# Define your domain object, you can couple it with Django if they are from DB
class Company(DjangoObjectType):
class Meta:
model = dm.Company
class CompanyQuery(graphene.ObjectType):
companies= graphene.List(graphene.NonNull(Company), required=True)
company = graphene.Field(Company)
def resolve_company(self, info, **kwargs):
....
return dm.User.objects.get(user_id=user_id)
def resolve_companies(self, info, access_group_id=None):
model = dm.Company.objects
....
return model.filter(....)
## include into the Query
class Query(
CompanyQuery,
....
):
pass
## include different types
schema = graphene.Schema(query=Query, ....)
It starts with the code, to define the domain object and then the service how to resolve that instance, then will generate the contract people can reach:
For Ariadne, it will start the other way around, so we will first define the schemas:
## define the domain object
type CompanyInfo {
name: String!
address: [String!]!
staff: [Satff!]!
owner: String!
....
}
## define the resolver
type Query {
companies: [CompanyInfo!]!
....
}
## Then we load the schema
schema = load_schema_from_path("...graphql")
## Then link the schema with the resolver
make_executable_schema(
schema, company_resolver()....
)
## define the resolver
company = ObjectType("CompanyInfo")
@company.field("address")
def resolve_address(
_: None, info: GraphQLResolveInfo, ....
) -> FrameworkInstance:
return get_address(....)
Both routes will reach the same destination:
Now come with the debate: some prefer the Ariadne approach, claiming it’s for better decoupling.
However, looking back at history, this is the same debates happened many times:
We have this debate with web services, between contract first vs contract last, WSDL to code vs Code to WSDL.
We have this debate with ORM, whether domain first or last.
But what has happened? Both approaches survived through time.
Actually similar to DI, a lot of developers are aware of the decoupling writing the components through separate configuration files (XML). However, tight coupling wins out, we are almost using annotation everywhere.
The claim of schema first or contract first, so that it could be decoupled and have a single source of truth, actually worked out with code first as well. Look at what happens at REST services. We start with code first, write the services, then leave out to swagger to expose the single source of truth (the contract).
Ultimately, I think it boils to the maturity of the tooling. If either side has supreme tooling over the other side, it will win out regardless of concepts.
This is no requirements on the frameworks, developers can choose whichever prefered either plain javascript, react, angular or vue. I have been building extensions with jquery, angular and react. As long as the final package can be rendered as normal web page, it will work with chrome extension.
Set up the manifest
We need to have the manifest set up. Here are some of the configurations:
most importantly, we need to have the background, popup, options and content script configured as needed. To invoke the extension by keyboard, the browser and page actions need to be set up.
Build and Test out
You can load the extension locally using the `developer mode`
Note: if you are using react and react-scripts for packaging, you need to set up for the CSP policy:
Publish to the market
You need to create a developer account and publish using the chrome developer console.
Service Mesh is a design to provide an infrastructure as a service layer within the cloud service. Sidecar is one example of such implementation.
SideCar as its name is acting as a decoupled component attached to other microservices to cater to those cross-cutting concerns. One example is a volume mount across microservices.
As mentioned in my other post, mount S3 as share drive, it’s a great feature to be able to access the S3 for CRUD operations. Some common use would embed the volume mount into each container or pod whichever needs the S3 access.
However, there is a security concern with this.
The ACL needed to mount the FUSE drive is `SYS_ADMIN` at a minimum. So to run this as a single container, we need to provide:
docker run --rm -it --cap-add SYS_ADMIN s3-sidecar bash
to run it with docker-compose:
s3-sidecar: restart: on-failure image: s3-sidecar init: true build: context: s3-sidecar target: dev environment: - DEPLOYMENt=STAGING privileged: true cap_add: - SYS_ADMIN # This is needed for mounting the volume
all these would expose the uplifted privilege to the container and pod. With this access, there are ways some experienced developers would be able to bypass the designated location /s3 for example in the above pod and write or delete other files within the VFS.
Unless the permission required for FUSE mounting is corrected, it’s important to segregate the component doing this mounting.
The implementation of this segregation is sidecar. So instead of embedding the mounting into each container or pod, we will create a dedicated sidecar to do the single point mounting. We will have different security control for this single component to disable it being exposed or exploited. While at the same time, for those containers or pods need to access S3, they will just have read-only access into the sidecar volume.
Here is the implementation:
For the sidecar, it will be provided with the needed permission `SYS_ADMIN` to run the mounting.
Note: the mountPropagation should be Bidirectional, as we need the access to be able to update the content back into S3
Note: we will limit the mountPropagation to HostToContainer. So that if write or update to the mount place or sub directory, they will be available in these containers. However, these containers shouldn’t propagate contents back into S3.
(base) ➜ lib git:(develop) ✗ python –version
dyld: Library not loaded: @executable_path/../.Python
Referenced from: /Users/jackie/.local/share/virtualenvs/streamlit-GdDAcdiW/bin/python
Reason: image not found
[1] 81477 abort python –version
(base) ➜ lib git:(develop) ✗ ..
dyld: Library not loaded: @executable_path/../.Python
Referenced from: /Users/jackie/.local/share/virtualenvs/streamlit-GdDAcdiW/bin/python
Reason: image not found
(base) ➜ streamlit git:(develop) ✗ make all-devel
dyld: Library not loaded: @executable_path/../.Python
Referenced from: /Users/jackie/.local/share/virtualenvs/streamlit-GdDAcdiW/bin/python
Reason: image not found
clear old virtualenv
(base) ➜ lib git:(develop) ✗ rm -rf pipenv --venv
(base) ➜ lib git:(develop) ✗ pipenv --venv
No virtualenv has been created for this project yet!
Aborted!
(base) ➜ lib git:(develop) ✗ pipenv --rm
No virtualenv has been created for this project yet!
Aborted!
recreate new env with specific version (exist on local OS)
(base) ➜ lib git:(develop) ✗ pipenv --python 3.7
Warning: the environment variable LANG is not set!
We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.
Creating a virtualenv for this project…
There are three kinds of ports widely used in kubernetes resource management, for example: port is the port exposed within the cluster. so other nodes/pods from same cluster can access it through service:port targetPort is the port exposed from within the pod. nodePort is the port exposed to outside world. so on public network, it could be accessed as public-ip:nodePort
AWS S3 is a popular choice nowadays for cloud storage. As Amazon claims:
Amazon S3 is designed for 99.999999999% (11 9’s) of data durability because it automatically creates and stores copies of all S3 objects across multiple systems.
There are common needs we would like to mount the cloud drive (as a shared drive or local disk). As such we can access the cloud drive (S3 bucket here) from other cloud services or even our local OS. It will be super convenient for viewing, writing and updating files in the cloud.
The tool I have leveraged here for mounting is called s3fs, which is to mount the S3 as the FUSE file system.
Here are the steps to achieve the state:
Create the S3 bucket
As a preliminary, we need to have the S3 bucket created.
There are two ways to create the bucket, either from the web console or using the AWS CLI.
Some prefer to use the web console, as it’s quite intuitive to access the console and create the bucket there. Most of the cases it’s just to click the create bucket button then follow the steps:
You will be able to review and make changes if needed before the bucket is created:
most of the cases you would like to Block all pubic access.
Note, when you create the bucket using the console, you need to select the region. Make sure you choose the region as close as possible physically to your server/computer location where you would access the bucket. These are the regions and corresponding codes at the moment of writing:
However, once the bucket created, you will note that:
Don’t get confused, as the S3 console will display all buckets regardless of regions. You can find the actual region the bucket is residing in the corresponding column.
Alternatively, if you are more familiar with the AWS CLI, you might like to create the bucket from the command line.
you would need to have your AWS CLI set up properly before you are able to run the command:
aws configure
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: us-west-2
Default output format [None]: json
after this, it will generate a valid config and credential file
~/.aws/config
~/.aws/credentials
set up the proper access
After the bucket is created, you need to set up the proper access. There are two approaches to set up for S3 access, either using s3 policy or through IAM policy.
Personally I think the IAM policy should be the defacto place for controlling most of the AWS resources access. The reason being it’s decoupled. It’s specifically for access control alone, while not tied to any specific role or buckets. While at the same time, S3 policy is bucket specific, which would work if your bucket is eternal.
However, you have both choices, you can make your own judgment here.
For IAM policy, you need to create the role, then associate the principal/person who needs to access the bucket with the role:
Note: you need to grant these access to the bucket
“s3:ListBucket”,
“s3:PutObject”, “s3:GetObject”, “s3:DeleteObject”
so in the policy, it would be something like this:
for the object access, make sure you grant to both resources
arn:aws:s3:::bucket arn:aws:s3:::bucket/*
you can use the AWS simulator for confirming your set up is correct with the access:
in most cases, especially if you would like to access the mount from other cloud services, you need to mount it by role. Normally those roles are tied to the cloud resources.
for example, if you would like to access it from an EC2 instance, you just need to grant the role to that EC2 instance would work.
mount by key
In most cases, you might not have the access key, as this normally it’s owned by system admin. But just in case if you do have, you can mount it using your AWS access keys:
As the image command is only run during build time, however, while running the container, we might need to access some environment or configuration variable. here is the workaround:
ARG variable=unknown ## Build time
ENV variable=${variable} ## Run time
Then to pass in the arg,
docker build--build-arg version=0.0.1
for docker-compose, then
container:
image: image
restart: always
build:
context: dockerfile
args:
version: ${version} ## alternatively, default a value here
Upgrade your chrome experience. Leverage on this app to handle tabs and urls navigation.
You can maximize your screen space, use Chrome in complete full screen mode by dropping off the address bar and the tab bar.
* type a URL followed by enter to visit the specified address
* type a term followed by enter to search the term (the search engine is configurable)
* type a term to navigate to any popular search suggestions
* type a keyword to navigate to any mapped URL (the map is configurable)
* type a term to navigate to any open tabs
* type a term to get the likely result you would like to visit
**Configurations**
change the search engine from the options page
configure the keyword and URL mapping
**Keyboard Shortcut:**
Windows: Alt+L
Mac: ⌥+L
Full Screen your chrome, then leverage ont this plugin for URL box
Enjoying Chrome in Completely Full Screen mode even without omnibox (Zen). Leave the address typing to ZenHelper.
Keyboard Shortcut:
windows: Ctrl+Shift+L
mac: Cmd+Shift+L
Note: This is still beta stage, new feature WIP.
After all its technology both still in exist, and the exist is with a reason. It’s ultimately boil down to personal preference, but neither should be forced as superior.
it’s not a completely novel requirement to get the status for asynchronous processing, for example, to get the file upload progress, to get the order status, alert for processing failures etc.
IMO, there are mostly two approaches for it,
1.synchronous
whether it’s through REST/GraphQL/Queue/DB/FS, if the client is polling for status, that’s a synchronous call, which kind of patched for the original fire and forget async process.
asynch
this would still be a patch for the original async process, however, it’s really valid. since it’s a “new” requirement from the client to get the status. instead of polling, which almost removed the merits of the original async call, the client could expose a webhook for the server to post back status asynchronously. as such, even though it’s a patch, it will be patched async, leave both processes fire and forget.
recently i have a need to build/start/stop some sibling containers (vs docker within docker), the way to do it is to expose a pipelien from the host to the container:
for single container:
docker run -v /var/run/docker.sock:/var/run/docker.sock
history is full of similarity.
this is such a repeat of WSDL2Java vs Java2WSDL debate. personally, it’s nothing black or white, right or wrong, similar to the ORM vs mybatis, it’s just opposite design suitable for individual use cases.
it might be true that Google or Facebook has some monorepo alive since more than a decade or two decades ago, however, that doesn’t means it’s the right approach for any new projects to adapt in 2019.
tech has been improving on loose coupling and SoC, which is a design idea since GOF and even earlier. this has continuously generating the benefits from version control, project segregation, building, CI/CD and micro services, which resulted faster and better quality delivery and long term maintenance. I will be surprised if monorepo would flourish in next decade or any near future.
quite interesting to see all those ideas/noises/debates come back and forth though.
it’s a dilemma, the reason why the language become more popular, (especially among the new or maybe amateur programmers) is due to its loose design, easy to learn (at the cost of not to leverage on the static typing, multi-threading concurrency, and JIT and AOTs for example), which is the exact reason for its being not as performing as the other languages.
Its been around a decade since MS tried to emulate the success of java, ever since j++ to. Net c#.
Now
MS is a java shop now.
Java is not only de facto langue for thoughtful enterprise app, its ability to keep evolving from several years release cycle to half yearly, and with so many version poped up each with up to date features design principal, yet as the same time as backward compatible as possible (its unimaginable of the python versioning issue from a java world.) java is very mature stable language yet growing at even faster speed than any new lanagues with the very solid design carefully thought of language principle, its able to grow even faster and at the same time as always stable and trustworthy.
Years back, it was very much a hassle and careful thought process to build the right projet skeleton with the right code structure and inline libraries. After that it’s another equally if not more challenging to pave the runway for the poc to build test deploy and put on suitable scale of Web servers/ containers, application server, and databases with normalisation and vertical horizontal scaling.
All these efforts have now significantly saved thanks to all new frameworks which bundle all these together, providing an optioned suite.
With the popular of the frameworks and tools, then come with now a new challenges, which is to properly leverage on the tools.
Aws for example, has shoot up from single digit toolset size to now probably hundred.
Container has gone from vm chroot jail docker cluster k8s swam.
There is definitely value added to have the skills (DevOps or admin or infra) to pave the right platforms at this era.
The default way to authen then talk with registry is through
docker login.
The user name is aws and password could be retrieve using
Aws ecr get-token
So far it’s pretty straightforward.
However, there is a caveat there. The token from aws CLI is valid for 12 hours only, this is aws’s approach to secure the access, in case the token is compromised, it’s to be expired then only authorised could retrieve the new token.
One possible approach to keep the docker CLI work is to refresh the
Docker login
Every 12 hours. Which is not difficult however is very ugly.
Instead, aws has this Credential helper. So with the Aws-ecr-Credential-helper installed, when we run docker CLI, it’s able to pick up the config from ~/.docker/config.json
That it would leverage on the helper to talk to the specific ecr instance. And the helper in turn would leverage on pre-configured ~/.aws/credential & ~/.aws/config to pick up the right access key and secret etc to talk with ecr.
This is a cool solution not only for Docker CLI but actually a lot serverless platform as well which relies on containers.
Have been working on some serverless framework recently, which i have put onto EKS.
most of the stuff worked, except the cli, which leveraged on k8s client-go library to authen is not able to do so with EKS. (working well with Azure AKS and GCP).
turns out the issue was with k8s client-go library, which doesn’t deal with aws-iam-authenticator. as a work around, the patch is to apply the service account as a bearer token.
//command to get the token
kubectl describe secret account -n namespace | grep -E '^token' | cut -f2 -d':' | tr -d " "
then in the client-go, patch the token into the bearer header:
//retrieve the token either from secret file or env var
//token, err := ioutil.ReadFile("~/secrets/kubernetes.io/serviceaccount/" + v1.ServiceAccountTokenKey)
//token := os.Getenv("BEARER_TOKEN")
//add the header if its not yet there
r.headers.Set("Authorization", "Bearer xxx")
//before the real http call
resp, err := client.Do(req)
it’s not a perfect language, but it’s the one in the lead position and continuously approaching and being the most complete, feature rich and thoughtful language, suitable for most large entreprise grade build out and even with future support, expansion and scaling lookout in mind.
This crazy race has no winner at all since it is neverending. That’s it! Yesterday you were learning Backbone.js, jQuery, Knockout.js, Ember.js, then AngularJS and now ReactJS, Next.js, Vue.js, all Angular flavors. Today, up comes Ext JS and Aurelia, the new ones. And tomorrow another will come up. The framework array list is endless.
Personally, there is no perfect solution for all.
Nowadays, there are so many solutions/partial-solutions/broken-solutions, which for people, normally has a lazy tendacy, would like to fanatic easy to understand simply because it’s easy. But easy solution is normally suitable for special or specific domain, in some cases, its broken. Otherwise, for example, why the mostly widely used langue for enterprises is java?
