AWS Lambda + Proteus POC - Service Side Connectivity Question


#41

Thanks! I will try this.


#42

So I think I found an easy quick way to reproduce this with the quickstart sample.

Setup:

  1. Run Proteus on Docker on another box (for me its another laptop) (see broker args below)
  2. On my local Mac, I set a local /etc/hosts for 192.168.88.123 devbox
  3. I also setup my mac to use an ethernet (turned wifi off) so as to give me a way to simulate a broken connection.
  4. Set my client’s server and client to point to the broker via prop: netifi.proteus.broker.hostname= “devbox”
  5. Start the server on the IDE
  6. Start the client on the IDE (request should send and receive ok)
  7. Disconnect the cable for about 5 seconds or so and reconnect

Result:

  • Server just hangs as I’ve been describing. No log output.
  • However, the client does thrash a bit but it ultimately recovers once I restart the server
    • Once it does recover, it starts sending like crazy :slight_smile:

I actually modded the quickstart client to not be a CommandLineRunner but to continuously send messages but I think that’s not a factor here:

Slightly modded client

    @Scheduled(fixedRate = 10 * 1000)
    public void run() {
        // Create Request to HelloService
        HelloRequest request = HelloRequest.newBuilder()
                                   .setName("World")
                                   .build();
    
        logger.info("Sending 'World' to HelloService...");
    
        // Call the HelloService
        logger.info(client.sayHello(request).block());
    }

Broker Args

'-Dnetifi.broker.tcp.address=0.0.0.0' '-Dnetifi.broker.tcp.publicAddress=192.168.88.123' '-Dnetifi.authentication.0.accessKey=9007199254740991' '-Dnetifi.broker.console.enabled=true' '-Dnetifi.authentication.0.accessToken=kTBDVtfRBO4tHOnZzSyY5ym2kfY=' '-Dnetifi.broker.admin.accessKey=9007199254740991' '-Dnetifi.broker.admin.accessToken=kTBDVtfRBO4tHOnZzSyY5ym2kfY='

#43

Now, I’m curious about a cluster setup which would be more real-world so perhaps I will learn and try to set that up.


#44

Here’s what we use for testing with kubernetes - might help - you can also use consol for discovering other broker nodes. We’re working on adding support for EC2 tags, and depending on how you configure your cluster you could just DNS names.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: broker-sa
rules:
  - apiGroups: [""]
    resources:
      - endpoints
    verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: broker-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: broker-sa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: broker-sa
subjects:
  - kind: ServiceAccount
    name: broker-sa
    namespace: default
---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  labels:
    app: proteus
  name: proteus
  namespace: default
spec:
  template:
    metadata:
      labels:
        app: proteus
      name: proteus
    spec:
      serviceAccountName: broker-sa
      automountServiceAccountToken: true
      hostIPC: true
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      imagePullSecrets:
        - name: netifi-private
      containers:
        - name: proteus
          image: netifi.azurecr.io/proteus:1.5.3.BUILD-SNAPSHOT
          imagePullPolicy: Always
          env:
            - name: BROKER_SERVER_OPTS
              value: "$(BROKER_SERVER_OPTS) '-Dcom.sun.management.jmxremote.rmi.port=40000' \
                    '-Dcom.sun.management.jmxremote=true' \
                    '-Dcom.sun.management.jmxremote.port=40000' \
                    '-Dcom.sun.management.jmxremote.ssl=false' \
                    '-Dcom.sun.management.jmxremote.authenticate=false' \
                    '-Dcom.sun.management.jmxremote.local.only=false' \
                    '-XX:+UnlockCommercialFeatures' \
                    '-XX:+FlightRecorder' \
                    '-XX:ActiveProcessorCount=2'
                    '-Djava.rmi.server.hostname='$(BROKER_PUBLIC_ADDRESS)"
            - name: AUTHENTICATION.0.accessKey
              valueFrom:
                secretKeyRef:
                  name: authentication-token
                  key: accessKey
            - name: AUTHENTICATION.0.accessToken
              valueFrom:
                secretKeyRef:
                  name: authentication-token
                  key: accessToken
            - name: ADMIN_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: broker-admin
                  key: accessKey
            - name: ADMIN_ACCESS_TOKEN
              valueFrom:
                secretKeyRef:
                  name: broker-admin
                  key: accessToken
            - name: BROKER_PUBLIC_ADDRESS
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: WEBSOCKET_BROKER_PUBLIC_ADDRESS
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: ADMIN_PUBLIC_ADDRESS
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: CLUSTER_PUBLIC_ADDRESS
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: KUBERNETES_DEPLOYMENT_NAME
              value: "proteus"
            - name: KUBERNETES_PORT_NAME
              value: "cluster"
            - name: ENVIRONMENT
              value: "kubernetes"
            - name: SSL_DISABLED
              value: "false"
            - name: PROMETHEUS_BRIDGE
              value: "true"
          ports:
            - name: metrics
              containerPort: 8888
              hostPort: 8888
            - name: cluster
              containerPort: 7001
              hostPort: 7001
            - name: tcp
              containerPort: 8001
              hostPort: 8001
            - name: websocket
              containerPort: 8101
              hostPort: 8101
            - name: admin
              containerPort: 6001
              hostPort: 6001
            - name: console
              containerPort: 9000
              hostPort: 9000
            - name: jmx
              containerPort: 40000
              hostPort: 40000
          resources:
            limits:
              cpu: "2"
              memory: "2Gi"
            requests:
              memory: "2Gi"
              cpu: "100m"
---
apiVersion: v1
kind: Service
metadata:
  name: proteus
  namespace: default
spec:
  selector:
    app: proteus
  type: LoadBalancer
  ports:
    - name: metrics
      port: 8888
    - name: cluster
      port: 7001
    - name: tcp
      port: 8001
    - name: websocket
      port: 8101
    - name: admin
      port: 6001
    - name: console
      port: 9000
    - name: jmx
      port: 40000

#45

Thanks - I think I know where to look now. I think this is a bug. Going to setup the same test using a different machines…


#46

Setting io.netifi.proteus to info will be helpful too - it will print information about connecting, and reconnecting. <logger name=“io.netifi.proteus” level=“info” additivity=“false”> That’s probably why its not printing anything.


#47

Hi,

I was able to re-produce the issue myself using your example, and figured out what the problem is. The keep alive should be enabled by default. There was a bug in the code causing the Proteus-java client it to ignore all keep alive settings and disable it. I fixed the java client to use the client alive settings, and now the RSocket connection properly detects when a network connection is broken, and now re-connection logic kicks in properly. The fix for this is proteus-java 1.5.5

Thanks,
Robert


#48

Well, look at that! Awesome! Thanks!


#49

After a whole day, now we’re cooking with gas. Appreciate the quick support!