Both client and server service

Hi,

I have created service which exposes API over RSocket-RPC. In some cases this service must call neighbour same service to retrieve some data, i got
io.netty.util.IllegalReferenceCountException: refCnt: 0

Where should i research?

Hi,

What versions of rsocket-rpc and rsocket-java are you using?

Thanks,
Robert

Hello!

Dependencies are:
-com.google.protobuf:protobuf-gradle-plugin:0.8.10

  • com.google.protobuf:protoc:3.9.1
  • io.rsocket.rpc:rsocket-rpc-core:0.2.17
  • io.rsocket:rsocket-core:0.12.1
  • io.rsocket:rsocket-transport-netty:0.12.1

Thanks.

There is additional info.

The exception with refCnt: 0 fixed.

So, lets describe the problem deeper. There is service named ‘gateway’ and service named ‘processor’. The gateway service provides external API for other users (REST for the current time) and when you call him the gateway calls ‘processor’. Processor service has some replicas (3 for example) and in some cases ‘processor’ has the requested data and sometimes not. When ‘processor’ hasnt data he calls another replica (in this moment processor has already knew where is the data we need), retrieves data and returns to ‘gateway’ and ‘gateway’ returns to user. It works well over http and i want to migrate internal calls to RSocket-RPC. So, exeption was thrown in case when processor needs to call another replica of him: Gw -> Pr1 -> Pr2

Common actions:

  • added project for IDL where service described (two proto files, one for DTOs and one for service description).
  • generated classes (everything is well in this moment)
  • added implementation to Gw and Pr-s

Everithing works well except when one replica of ‘processor’ needs to call another replica.

Processor has server bean (RSocketFactory.receive(). ...) to handle requests from gateway.
Bad solution with refCnt: 0 was (inside ‘processor’ service): create rsocket-client bean to make calls to another replica. In this default case server bean recieves request from gateway with some metadata and client-bean is trying to call another replica with same metadata, in the result there is refCnt: 0.

The solution is: make proxy-object for this interaction with overrided methods, which call another replica without using incoming metadata from gateway-call.

But there is another problem.

Processor service always crashes when starting to interact with another replica. The case is:

  • User calls gateway (REST)
  • Gateway calls processor (RSocket-RPC)
  • Processor handles request and see that he hasnt data
  • Processor trying to create client-bean to another replica (bean creates well)
  • Processor trying to interact with another replica over client-bean from previous step (RSocket-RPC)

When the interaction starts, the processor-instance crashes (Pr1 from this example Gw -> Pr1 -> Pr2)

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa6f1069a7e, pid=6830, tid=0x00007fa6b09c5700
#
# JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build 1.8.0_222-8u222-b10-1ubuntu1~16.04.1-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# v  ~BufferBlob::jni_fast_GetIntField

The last logs i saw in Pr1:

  • receiving -> Frame => Stream ID: 15 Type: REQUEST_RESPONSE Flags: 0b100000000 Length: 65 Metadata: (metadata from GW request) and Data: (data from request)
  • then, i see how Pr1 is trying to call Pr2 sending -> Frame => Stream ID: 1 Type: REQUEST_RESPONSE Flags: 0b100000000 Length: 58 Metadata: (request meta) and Data: (request data)
  • Next lines A fatal error has been detected ... as described above

UPD: Java 12 - same problem

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f9748bdd39f, pid=16675, tid=16887
#
# JRE version: Java(TM) SE Runtime Environment (12.0.2+10) (build 12.0.2+10)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (12.0.2+10, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# v  ~BufferBlob::jni_fast_GetIntField

@maksym can you take a look at this as you’ve been working the gateway recently?

This appears to be general RPC problem - looks like OP does not use netifi-http-gateway, but some custom implementation (@shpall can you confirm? )

In case of Gw -> Pr1 -> Pr2

There is no ‘netifi-http-gateway’ on GW, its a service with webflux and RSocket-RPC client. It handles REST and makes requests over generated rsocket-client to Processor. Pr1 and Pr2 are two replicas of Processor. Each of them has generated rsocket-server to handle requests from Gw. In this moment all is ok.

JVM crashes when Pr1 creates a new (generated) rsocket-client (same as inside in Gw) to Pr2 (and when pr1 is trying to communicate with pr2)

Are you trying to dynamically generate clients to create rest to rsocket-rpc calls?

Hello.

Yes, i’m generating clients dynamically. I found the solution and i will update this topic with more info later.

1 Like