Separate your worker process in your Go application

Separate your worker process in your Go application

Golang has an awesome feature called Goroutine AKA known as a lightweight thread managed by Go runtime. But most of the time use this for our async task.
Like
When we have a function called orderCompleted. And after the order is completed function need to do another task to generate an invoice, send a notification, send an SMS/email, and push data to the mix panel or WebEngage so one.

Also, we often use goroutine for task management. Like in your go web application, in your main function, you serve an HTTP application by calling
err := http.ListenAndServe(":3333", mux) this method. What this function will do. This will continually listen to the port 3333 and serve the content if anyone calls. So this function will never stop until you stop gracefully.

But what if you need to add another process to work the same way?
What if you have to listen to another port within the same application? What if you need to run rabbitMq consumer to continually listen? What if you need to run SQS consumers to continually listen?

For this situation, we all try to use the goroutine right?

THIS IS ACTUALLY WRONG….. AND YOU WILL SUFFER

suffering

Look when your goroutine will run all your working processes in the same main function you will not notice the problem instantly but you will someday. Here I am telling you why

The single responsibility principle

Your main function is responsible for doing only one job which is listening to port 333. But you tell your function to listen to the MQ consumer in different threads, listen to the Sqs consumer in a different thread, listen to the Kafka consumer in a different thread, and so on.

For this reason when any error occurs MQ consumer listening function. Your whole HTTP application might crash or stop.

If you are running your application in any popular orchestrate tools like Kubernetes or Docker Swarm. You might add your policy to restart automatically when the app stop. But the starting moment might be too long and eventually, you might lose your reputation.

But It shouldn't happen that right?

Why the hell Error in MQ consumers shutting down the whole system?

But If I say instead of doing this you can do this that way

Separate your process and run it separately

If we separate this process into separate main functions and run those main functions separately. So that even if we face any trouble in the MQ consumer process the HTTP server function will stay alive.

And if you are using any orchestrate tools you can separate them into different pods. like

app-api-server-pod // for HTTP process
app-mq-worker-pod // for mq consuming process
app-sqs-worker-pod // for sqs consuming process
app-kafka-worker-pod // for kafka process

You can also get another benefit for doing this, you can limit your worker pod which you can't do before.

Thanks and Happy Coding :)