Performance. Errors. Logs. One Tool. Everything Developers Need to Support Their Apps

Stackify Blog

Subscribe to Stackify Blog: eMailAlertsEmail Alerts
Get Stackify Blog via: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Blog Feed Post

Message queues & you – 12 reasons to use message queuing

At Stackify we receive a lot of data from your apps to our Retrace APIs. We queue all of that data as soon as it gets to us and then we use separate background services to process the data. Message queues help a lot with ensuring data is never lost, traffic spikes, etc. We decided we to put together a list of why you should be using message queues!

1. Redundancy via persistence

Redundancy is one of the most obvious advantages to message queues. Application crashes, timeouts, errors in your code, and other problems are just part of the norm. This is especially true in applications that process millions or billions of transactions per month.

Queues helps with redundancy by making the process that reads the message confirm that it completed the transaction and it is safe to remove it. If anything fails, worst case scenario, the messages is persisted to storage somewhere and won’t be lost. It can be reprocessed later.

2. Traffic spikes

You don’t always know exactly how much traffic your application is going to have. For example, at Stackify we receive billions of messages a month. We have no way to know what our clients are going to send us. By queuing the data we can be assured the data will be persisted and then be processed eventually, even if that means it takes a little longer than usual due to a high traffic spike.

3. Improve web application page load times

Queues can be useful in web applications to do complex logic in a background thread so the request can finish for the user quickly. If someone places an order on your website, that could involve a lot of different things that have to happen. You can do the minimum of them and return success to your user and kick off the rest of them in a background thread to finish up, without using a full message queuing system and background apps. Most programming languages have ways to do this now. Examples: Resque, Hangfire, etc.

4. Batching for efficiency

Batching is a great reason to use message queues. It is much more efficient to insert 100 records in to a database at a time instead of 1 at a time, 100 times. We insert a lot of data in to elasticsearch and SQL Server. Batching helps us optimize their performance by tuning the size of the transactions.

5. Asynchronous messaging

Queues can be great in scenarios where your application needs something done but doesn’t need it done now, or doesn’t even care about the result. Instead of calling a web service and waiting for it to complete, you can write the message to a queue and let the same business logic happen later.

6. Decouple by using data contracts

By using a queue between different parts of your software you can decouple the hard dependencies. The format of the message in the queue becomes your data contract and anything that knows how to read that message format can be used to process the transaction. This could be useful for parts of your code that are even written in different programming languages.

7. Transaction ordering and concurrency challenges

If 1000 people are placing an order on your website at one time, that could create some problems with concurrency and ensuring that the first order in, finishes first. By queuing them up, you could then guarantee their order and control how many are even processed concurrently.

8. Improve scalability

Message queues enable you to decouple different parts of your application and then scale them independently. Using Azure, AWS, or other hosting solutions you could even dynamically scale that background service based on CPU usage or other metrics. Message queues can help a lot with scalability and elasticitiy.

9. Create resiliency

By breaking your app up and separating different components by queues you inherently create more resiliency. Your website can still be operational even if part of the back end processing of the order is slightly delayed. At Stackify we design our incoming APIs to do as little as possible beyond queuing the data to ensure that nothing else can bring down the ability to receive data. Even if SQL Server is down, we want to be able accept data!

10. Guarantee transaction occurs once

Message queues are designed to be transactional. When you read a message off of a queue, you have to tell it where you completed processing the message before it is completely removed from the queue. This helps to ensure that a transaction only occurs once.

11. Break larger tasks in to many smaller ones

Another good use for queues is breaking up a larger task in to lots of little pieces and then queuing them all up. We have a good example of this at Stackify. If you edit a monitoring template for a group of servers, we then need to update the monitors on every server that uses that template. We can can queue up a message for every server and potentially do them concurrently as smaller operations.

12. Monitoring

Message queuing systems enable you to monitor how many items are in a queue, the rate of processing messages, and other stats. This can be very helpful from an application monitoring stand point to keep an eye on how data is flowing through your system and if it is getting backed up.

Does every app need message queues?

No, of course not. Like most things in life, there is a place for everything.

However, I would suggest at least thinking about improving the performance of your web applications by using background tasks to offload some long running transactions.

Alternative “In app” background processing

You can consider Hangfire, Resque and other similar projects for this. Those projects act as sort of mini queuing systems and can persist the jobs for resiliency. They run entirely in your web application so it doesn’t require standing up additional servers or anything.

Another option without the resiliency is using fire and forget tasks in your code. But if your app crashes before it finishes or some sort of error occurs, that operation would be lost.

ASP.NET has a handy way to do this. If you do it this way, the framework will even delay stopping your app until your code finishes. But if your app crashes, it would be lost. Using Hangfire would provide more functionality and the resiliency.

	HostingEnvironment.QueueBackgroundWorkItem(x =>
		//do some stuff here, like maybe send an email?
		SmtpClient smtp = new SmtpClient();

You can read more about HostingEnvironment.QueueBackgroundWorkItem at MSDN.

Message Queues Conclusion

Message queues a very critical component to our platform at Stackify. We queue and process billions of messages a month. For us, it would be hard to imagine not having message queues!

It can be a lot of work to stand up RabbitMQ, MSMQ or some other message queuing service. Although, Azure and AWS make it easier with their hosted offerings. We use Azure Service Bus ourselves and it has worked well for us.

How do you use message queues? Having any other great uses or tips? Comment below!

The post Message queues & you – 12 reasons to use message queuing appeared first on Stackify.

Read the original blog entry...

More Stories By Stackify Blog

Stackify offers the only developers-friendly solution that fully integrates error and log management with application performance monitoring and management. Allowing you to easily isolate issues, identify what needs to be fixed quicker and focus your efforts – Support less, Code more. Stackify provides software developers, operations and support managers with an innovative cloud based solution that gives them DevOps insight and allows them to monitor, detect and resolve application issues before they affect the business to ensure a better end user experience. Start your free trial now stackify.com