API Services Fail to Start Silently

Software Versions

OS: Linux
JVM: Oracle Corporation 1.8.0_161 AMD64
Version: 4.7.4
Code: 18260

Expected behavior

An exception should be thrown and the node service should exit when the API services fail to start.

Actual behavior

Node starts with no API services and no exception about it thrown.

Steps to reproduce the behavior

Make gRPC service port 50051 occupied in advance
Start the node
Try to access the API endpoint

Backtrace

14:54:41.976 ERROR [rpc] RpcApiService
java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:50051
	at io.grpc.netty.NettyServer.start(NettyServer.java:328)
Caused by: java.net.BindException: Address already in use
14:54:41.983 INFO [app] All api services started.

UNIMPLEMENTED: Method not found: protocol.Wallet/GetAccount
io.grpc.StatusRuntimeException: UNIMPLEMENTED: Method not found: protocol.Wallet/GetAccount
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)

@halibobo1205

@halibobo1205 In addition to api services, do other services also have such demands?

@zeusoo001

@zeusoo001, IMHO: external services should not silently fail to start, such as event-plugins.

external services :

TYPE	SERVICE	default	available	note
API	RpcApiService	⚠️ force-enabled	FullNode\SolidityNode	GRPC
API	RpcApiServiceOnSolidity	⚠️ force-enabled	FullNode	GRPC
API	RpcApiServiceOnPBFT	⚠️ force-enabled	FullNode	GRPC
API	FullNodeHttpApiService	✅ enabled	FullNode	HTTP
API	HttpApiOnSolidityService	✅ enabled	FullNode	HTTP
API	HttpApiOnPBFTService	⚠️ force-enabled	FullNode	HTTP
API	SolidityNodeHttpApiService	❌ disabled	SolidityNode	HTTP
API	FullNodeJsonRpcHttpService	❌ disabled	FullNode	JSON-RPC
API	JsonRpcServiceOnSolidity	❌ disabled	FullNode	JSON-RPC
API	JsonRpcServiceOnPBFT	❌ disabled	FullNode	JSON-RPC
metric	Metrics	❌ disabled	FullNode\SolidityNode	Prometheus
EVENT	EventPluginLoader	❌ disabled	FullNode\SolidityNode	event subscribe
NET	TronNetService	✅ enabled	FullNode	P2P
ZK-SNARK	ZksnarkInitService	✅ enabled	FullNode\SolidityNode	shield transaction
LOG	logabck	✅ enabled	FullNode	custom Configuration

Force-enabled services: RpcApiService, RpcApiServiceOnSolidity, RpcApiServiceOnPBFT, and HttpApiOnPBFTService need to be adjusted to optional before the fix.

Do you mean that if API services fail to start, they need to throw real-time exception errors instead of only reporting errors when accessing the interface? This may require adjusting the priority of each service startup. It would be great if the startup mechanism could be optimized and error feedback could be provided.

@lxcmyf

This may require adjusting the priority of each service startup.
@lxcmyf why it needs to be adjusted?

@halibobo1205

@halibobo1205
I would like to say that various services may be divided into many types according to their functions, but there are also similar services that can be classified under the same category. Real time startup errors can be classified by type, making the sorting of errors clearer.

@halibobo1205

@halibobo1205 Should you check whether some configured ports are occupied before the main thread starts ?

@halibobo1205

@halibobo1205 Should you check whether some configured ports are occupied before the main thread starts ?

@317787106 There is no need to do this, just throw the associated error. See besu how to deal with this:

  private void waitForServiceToStart(
      final String serviceName, final CompletableFuture<?> startFuture) {
    do {
      try {
        startFuture.get(60, TimeUnit.SECONDS);
      } catch (final InterruptedException e) {
        Thread.currentThread().interrupt();
        throw new IllegalStateException("Interrupted while waiting for service to start", e);
      } catch (final ExecutionException e) {
        throw new IllegalStateException("Service " + serviceName + " failed to start", e);
      } catch (final TimeoutException e) {
        LOG.warn("Service {} is taking an unusually long time to start", serviceName);
      }
    } while (!startFuture.isDone());
  }

@lxcmyf

Real time startup errors can be classified by type, making the sorting of errors clearer.

@lxcmyf I don't think it's necessary, the error is even less clear, and the original error is still needed, you can see the sample above.

An exception should be thrown and the node service should exit when all external services fail to start.

Working on this.

@317787106

An exception should be thrown and the node service should exit when the API services fail to start.

Is it necessary to exit the node service? PTAL @317787106 @lxcmyf @tomatoishealthy

At present, some products I have used will not interrupt the main process when encountering some non-core service exceptions. Maybe TRON does not need to be forced to stop in this scenario?

But this is a very open topic, and it is also reasonable to stop the service when an exception is thrown.

I'd probably prefer to keep the service running, but I'd be fine with either.

@tomatoishealthy

Final decision: exit the node service when API services fail to start. cc @tomatoishealthy

Related PRs:

The following scenarios node also exit

The node is witness, but no private key is configured.
The configured rate limiter is not working as expected.
Configured auto-stop block time parsing error.

Navigation Menu

Repository navigation

API Services Fail to Start Silently #5820

Software Versions

Expected behavior

Actual behavior

Steps to reproduce the behavior

Backtrace

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

API Services Fail to Start Silently #5820

Description

Software Versions

Expected behavior

Actual behavior

Steps to reproduce the behavior

Backtrace

Activity

zeusoo001 commented on May 13, 2024

halibobo1205 commented on May 13, 2024

halibobo1205 commented on May 15, 2024

halibobo1205 commented on May 15, 2024

lxcmyf commented on May 16, 2024

halibobo1205 commented on May 16, 2024

lxcmyf commented on May 17, 2024

317787106 commented on May 17, 2024

halibobo1205 commented on May 17, 2024

halibobo1205 commented on May 17, 2024

halibobo1205 commented on May 23, 2024

halibobo1205 commented on May 24, 2024

tomatoishealthy commented on May 24, 2024

halibobo1205 commented on May 30, 2024

halibobo1205 commented on Mar 11, 2025

halibobo1205 commented on Mar 11, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions